You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've noticed an odd but somewhat nasty issue that appears to be reproducible. With Circuit Playground Express (but perhaps other M0 express and maybe M0 basic boards) if my project uses a lipo battery that runs low enough to brownout and turn off, after I swap in a new battery the entire SPI flash is erased and my old program is gone. It's happened twice in a row with my arc reactor project, and I have a feeling it always happens when power goes low like this (I've only ever run the battery completely dead twice and both times the FS was destroyed on next power up). My suspicion is in the low voltage or brownout state perhaps the SPI flash chip isn't readable and the board brownout resets, fails to read the flash, and assumes the FS is bad and erases it all (or maybe the FS was actually corrupted during the brownout).
Opening this as an issue for now to investigate further. I think we want to setup a board with a bench supply to VBAT and start it at 3.3 volts, then lower it bit by bit into brownout mode (probably around 2.5 volts or so) and see if it repros for easier investigation. I'm not sure there's a fix but if we can understand what's happening better perhaps there are workarounds or ways to prevent the FS from being destroyed.
I have a simple fix that won't reset the filesystem if it reboots into safe mode. I did try to reproduce this with a bench supply, and only managed to make it happen a couple of times, erratically. I also wrote a high-duty-cycle blinky program on CPX that lit all the Neopixels, and ran a LiPo down, but could not get it to erase. @tdicola when you're back I could pass you a .uf2 with the fix and you can test it with the arc reactor project.
@tdicola Pull request #319 would fix this, I think. I was able to break the filesystem a few times with the bench supply, and then after #319, I couldn't, but it wasn't really a good test. If you see this problem again after building the latest 2.x or using the upcoming 2.0.1 release, please reopen.
Using cpx 2.1.0, I have the same problem very reproducible with multiple units. In my case the fs is wiped reliably by loosening and wiggling the Lipo battery connector to produce an intermittent contact.
I had the same problem with a feather m0 express once, when I didn't remove the USB plug all the way.
I wonder if it could be a quicker and more convenient way of erasing flash than having to flash a special firmware for it.
I observed this occurring multiple times while we were doing demoing the TR808 drum (with fruit attached). In order to recalibrate, we were un-plugging and re-plugging in a 3.7V lipo battery.One of our kid helpers was leaving the lipo plug loosely in the socket, so that she could quickly disconnect and reconnect. I suspect that it was the quick disconnection and reconnection of power that may have contributed to this. I can re-create the issue by replicating the the loose socket quick switching of power. Later I connected a JST push-button switch to prevent this, and noticed that it eventually wiped again. This follows with what @BillyDonahue mentioned previously with 2.1.0.
Not sure if it's known or not, but this appears to be an issue still with 2.2.4 as well.
My setup is the development kit, with the 3 AAA batteries, battery case and CPX. Often enough if the battery switch is shut off, the code is lost and filesystem is reformatted when the battery switch is turned back on.
Huh, I wonder if its reproducible with a AAA switch because there is bounce on the switch.
@croadfeldt Could you save file, load an arduino sketch and then switch it on and off. Its still not clear to us if this is a CircuitPython issue or a hardware/SPI flash issue.
Raising brownout voltage limit to 2.77V did not fix the problem. After testing by turning various things off, adding delays, etc., I think that the usual reason for this problem is the boot_out.txt write that happens on every hard reset. If the write gets interrupted by a power fail (or reset?), the filesystem is damaged and will be re-created on the next startup. If I turn off the boot_out.txt write, I cannot reproduce the problem. If it's on, bobbling the power will cause it to happen in short order.
Usually what's written to boot_out.txt is the same every time. It contains mostly version information. It only changes if there's actually a boot.py that prints out something different, or the version of CircuitPython changes.
So we could check to see whether the file contents is identical what we want to write, and skip the write in that case, which would be most of the time. I'll try this out.
We can't really delay writing the contents of boot_out.txt, because when USB comes up, the filesystem is then usually set to be read-only to CircuitPython, and it's too late to write the file.
started w/ circuitpython 2.2.4 - running code.py - no usb connection, flop power switch on battery pack quickly results in wiped filesystem with just boot_txt
compiled code running raw - no usb connection, flop power switch has no effect on code, operation
redploy of circuitpython 2.2.4 - reinstall of code.py on filesystem, FWIW this persists through compiled code push from arduino IDE, this was a neat surprise when I put the 2.2.4 CP back on and my files re-appeared.
Here is a test version of CircuitPython: cpx-2v7-no-boot-out.uf2.zip. The .uf2 file is in a .zip file because GItHub requires it.Could you unzip it and try this version? It will refuse to work if the battery voltage is <2.8v (instead of 1.7v). It also does not write boot_out.txt when it starts up. If you still see a boot_out.txt, it is old. If the filesystem is erased, CIRCUITPY will be completely empty on startup. Try this test version with the battery switch as you have before, and see if your own files get erased or not.
If you have a multimeter, could you measure the voltage (under load) of the current set of batteries in the battery holder? The easiest way to do this is to measure the voltage across the VOUT and GND pins on the CPX (with USB not attached). Those pins are on either side of the battery socket. I want to know if the current batteries you're using are kind of weak. I have the same battery holder, and with a brand new set of batteries I measure 4.48V. With some good batteries I am getting 4.35V. It's a little less than 1.5V*3 or so because there's a protection diode that drops the voltage about 0.2 or 0.3V. It's not bad if the batteries are a bit run down -- we need to handle that case.
Was able to test this quick this morning, in testing 2v7, I was not able to reproduce the filesystem wipe. Put 2.2.4 back on, wiped the filesystem within 30 seconds.
Will be able ot test with a multimeter later tonight.
@croadfeldt Here is a more sophisticated update: cpx-2v7-smart-boot_out.uf2.zip. This file will write boot_out.txt only if the version information has changed or there is a boot.py file or equivalent, and it waits 1.5 seconds before writing the file. So the first time you use this, it will write the file (after 1.5 seconds), and then after that won't write the file, because it won't be different. Thanks for all the testing!
@tannewt@dhalbert - you're welcome! Thanks for your work to make this all work so well. :)
More results:
cpu-2v7-smart-boot_out does NOT wipe the filesystem. boot_out populates if the file is missing on startup as well. So both seem to be as expected, no wipe and boot_out populates.
Voltage is 4.678 under moderate load, moderate being all 10 LEDs up and running. No LEDS is 4.8v.
Putting 2.2.4 back on and I can now cause a FS wipe in under 5 seconds, lots of practice. 👍
Activity
dhalbert commentedon Oct 4, 2017
I have a simple fix that won't reset the filesystem if it reboots into safe mode. I did try to reproduce this with a bench supply, and only managed to make it happen a couple of times, erratically. I also wrote a high-duty-cycle blinky program on CPX that lit all the Neopixels, and ran a LiPo down, but could not get it to erase. @tdicola when you're back I could pass you a .uf2 with the fix and you can test it with the arc reactor project.
dhalbert commentedon Oct 9, 2017
@tdicola Patch is in branch
2.x_issue_280_brownoutin my repo https://github.com/dhalbert/circuitpython/. I have been fiddling with the voltage on a CPX but have only gotten it to wipe the flash once, and I can't replicate reliably.dhalbert commentedon Oct 12, 2017
@tdicola Pull request #319 would fix this, I think. I was able to break the filesystem a few times with the bench supply, and then after #319, I couldn't, but it wasn't really a good test. If you see this problem again after building the latest 2.x or using the upcoming 2.0.1 release, please reopen.
BillyDonahue commentedon Dec 4, 2017
Using cpx 2.1.0, I have the same problem very reproducible with multiple units. In my case the fs is wiped reliably by loosening and wiggling the Lipo battery connector to produce an intermittent contact.
deshipu commentedon Dec 4, 2017
I had the same problem with a feather m0 express once, when I didn't remove the USB plug all the way.
I wonder if it could be a quicker and more convenient way of erasing flash than having to flash a special firmware for it.
dhalbert commentedon Jan 22, 2018
jafine on discord reports a CIRCUITPY erase with 2.2.0, perhaps due to a low battery.
jafine commentedon Jan 23, 2018
I observed this occurring multiple times while we were doing demoing the TR808 drum (with fruit attached). In order to recalibrate, we were un-plugging and re-plugging in a 3.7V lipo battery.One of our kid helpers was leaving the lipo plug loosely in the socket, so that she could quickly disconnect and reconnect. I suspect that it was the quick disconnection and reconnection of power that may have contributed to this. I can re-create the issue by replicating the the loose socket quick switching of power. Later I connected a JST push-button switch to prevent this, and noticed that it eventually wiped again. This follows with what @BillyDonahue mentioned previously with 2.1.0.
croadfeldt commentedon Apr 30, 2018
Not sure if it's known or not, but this appears to be an issue still with 2.2.4 as well.
My setup is the development kit, with the 3 AAA batteries, battery case and CPX. Often enough if the battery switch is shut off, the code is lost and filesystem is reformatted when the battery switch is turned back on.
tannewt commentedon May 1, 2018
Huh, I wonder if its reproducible with a AAA switch because there is bounce on the switch.
@croadfeldt Could you save file, load an arduino sketch and then switch it on and off. Its still not clear to us if this is a CircuitPython issue or a hardware/SPI flash issue.
croadfeldt commentedon May 2, 2018
@tannewt Will do later this week when I have some time again.
tannewt commentedon May 2, 2018
Thanks @croadfeldt! @dhalbert is looking into the brownout detector settings now too.
dhalbert commentedon May 3, 2018
Raising brownout voltage limit to 2.77V did not fix the problem. After testing by turning various things off, adding delays, etc., I think that the usual reason for this problem is the
boot_out.txtwrite that happens on every hard reset. If the write gets interrupted by a power fail (or reset?), the filesystem is damaged and will be re-created on the next startup. If I turn off theboot_out.txtwrite, I cannot reproduce the problem. If it's on, bobbling the power will cause it to happen in short order.Usually what's written to
boot_out.txtis the same every time. It contains mostly version information. It only changes if there's actually aboot.pythat prints out something different, or the version of CircuitPython changes.So we could check to see whether the file contents is identical what we want to write, and skip the write in that case, which would be most of the time. I'll try this out.
We can't really delay writing the contents of
boot_out.txt, because when USB comes up, the filesystem is then usually set to be read-only to CircuitPython, and it's too late to write the file.(tagging @ladyada)
dhalbert commentedon May 3, 2018
This will also save wear and tear on the flash.
croadfeldt commentedon May 3, 2018
@dhalbert @tannewt
Results
started w/ circuitpython 2.2.4 - running code.py - no usb connection, flop power switch on battery pack quickly results in wiped filesystem with just boot_txt
compiled code running raw - no usb connection, flop power switch has no effect on code, operation
redploy of circuitpython 2.2.4 - reinstall of code.py on filesystem, FWIW this persists through compiled code push from arduino IDE, this was a neat surprise when I put the 2.2.4 CP back on and my files re-appeared.
dhalbert commentedon May 3, 2018
@croadfeldt Could you try a couple of things:
Here is a test version of CircuitPython: cpx-2v7-no-boot-out.uf2.zip. The .uf2 file is in a .zip file because GItHub requires it.Could you unzip it and try this version? It will refuse to work if the battery voltage is <2.8v (instead of 1.7v). It also does not write
boot_out.txtwhen it starts up. If you still see aboot_out.txt, it is old. If the filesystem is erased, CIRCUITPY will be completely empty on startup. Try this test version with the battery switch as you have before, and see if your own files get erased or not.If you have a multimeter, could you measure the voltage (under load) of the current set of batteries in the battery holder? The easiest way to do this is to measure the voltage across the VOUT and GND pins on the CPX (with USB not attached). Those pins are on either side of the battery socket. I want to know if the current batteries you're using are kind of weak. I have the same battery holder, and with a brand new set of batteries I measure 4.48V. With some good batteries I am getting 4.35V. It's a little less than 1.5V*3 or so because there's a protection diode that drops the voltage about 0.2 or 0.3V. It's not bad if the batteries are a bit run down -- we need to handle that case.
croadfeldt commentedon May 3, 2018
@dhalbert
Quick update.
Was able to test this quick this morning, in testing 2v7, I was not able to reproduce the filesystem wipe. Put 2.2.4 back on, wiped the filesystem within 30 seconds.
Will be able ot test with a multimeter later tonight.
Chris
tannewt commentedon May 3, 2018
Thank you for the help @croadfeldt !
dhalbert commentedon May 3, 2018
@croadfeldt Here is a more sophisticated update:
cpx-2v7-smart-boot_out.uf2.zip. This file will write
boot_out.txtonly if the version information has changed or there is aboot.pyfile or equivalent, and it waits 1.5 seconds before writing the file. So the first time you use this, it will write the file (after 1.5 seconds), and then after that won't write the file, because it won't be different. Thanks for all the testing!croadfeldt commentedon May 4, 2018
@tannewt @dhalbert - you're welcome! Thanks for your work to make this all work so well. :)
More results:
cpu-2v7-smart-boot_out does NOT wipe the filesystem. boot_out populates if the file is missing on startup as well. So both seem to be as expected, no wipe and boot_out populates.
Voltage is 4.678 under moderate load, moderate being all 10 LEDs up and running. No LEDS is 4.8v.
Putting 2.2.4 back on and I can now cause a FS wipe in under 5 seconds, lots of practice. 👍
dhalbert commentedon May 4, 2018
@croadfeldt ok, thanks! This is going into the next release. Your testing and your ability to break it have been very helpful 😃
Your batteries seem better than mine!
croadfeldt commentedon May 4, 2018
@dhalbert You're welcome, glad to be of help. I saw the note on PR #802, I see others are working as well. Awesome, thank you sir!
tannewt commentedon May 4, 2018
Thanks again everyone! Glad to see this much improved!