Raspberry Pi and Volumio 0.979 not responding

Hi!

I figured I’d give the new version 0.979-2016-08-20 a spin. Unfortunately it had issues like “connection reset by peer” etc if I tried to connect over HTTP.

SSH works, but the root partition is full(!)

If I log in fast enough, I can see that there is space, but repeated df -h / commands show that it’s going and du -sh /data/swapfile that it’s going there. Apparently this issue is preventing all usage.

I tried adding “noswap” to the boot parameters in /boot/cmdline.txt but this did not help anything. That’s because I couldn’t find any other references to a swap file anywhere usual like /etc/fstab…

What’s up with this?

It’s creating a swapfile since you’re using it on a PI with 512MB of RAM. Swapfile is necessary in this case. So leave it alone for 5 minutes after a fresh flash and you’ll see that everything is working .

Thanks @michelangelo for your reply, but unfortunately that didn’t help me too much :frowning:

It is true that my Pi doesn’t have an awful lot of memory, but having had it on for over five minutes, nothing has changed. Port 80 still does not reply:

volumio@volumio:~$ telnet localhost 80
Trying 127.0.0.1...
Trying ::1...
telnet: Unable to connect to remote host: Cannot assign requested address

volumio@volumio:~$ telnet 192.168.1.3 80
Trying 192.168.1.3...
telnet: Unable to connect to remote host: Connection refused

Same from outside the actual Pi.

The situation with regard to memory and disk space is this:

volumio@volumio:~$ free -m
             total       used       free     shared    buffers     cached
Mem:           482        444         37         16         17        323
-/+ buffers/cache:        103        378
Swap:          336          0        336

volumio@volumio:~$ du -sh /data/swapfile 
337M	/data/swapfile
337M	total

volumio@volumio:~$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
overlay         354M  346M     0 100% /

FWIW I think the partition was called something else before, but I tried also deleting the swapfile and rebooting, which did not help anything, but I think it may have changed the partition’s name. Will check tomorrow. Just to be sure: this experiment not affect the partition size or how it got filled up.

Could it just be that the overlay file system is too small or something?

Thanks again!

Yes, it was not resized on first boot, how big is your SD Card?
Could you try reflashing please?

Also, if telnet is not on, you would not be able to ping it :wink: and telnet is not enabled in volumio…

It’s a 4G card, and I tried reflashing, even with Etcher. Usually I go the dd route.

Is there anything in the flashing of the image that could affect the overlay partition’s resizing?

Telnet is just about transferring bytes over the internet, so opening a telnet to port 80 is useful for verifying the port
is up and allows you to type in the actual request:

GET / HTTP/1.1
Host: 192.168.1.3

and hopefully get a meaningful response.

So now that that’s out of the way, I found where the swap is created: github.com/volumio/Build/blob/m … dynswap.sh

The idea is nice, but contrary to common beliefs, swap is most useful when 1) it’s doled out in proportion to the speed of the disk and 2) it doesn’t grow beyond the size of the programs the kernel wants to swap out. Something like 512MB on a MicroSD is just brutal, because depending on the kernel
settings, it could swap out processes to have RAM to spare just in case, but Volumio doesn’t necessarily need it.

As a status report, I’m currently compiling an image that sets the swapfile to 64MB and hope it works ok, will let you know how it turns out.

Thanks!

Thank you for this comment, actually finding the ideal amount of SWAP will be very useful.

The best way to solve this issue would be finding a way to reduce the installation footprint, if you want to investigate in this direction that will be very very appreciated

(I’ll just mention here first, that the telnet test was flawed. For some reason I can’t connect to the http port on the Volumio Pi itself, even though the web UI works. This is irrelevant for the purpose of this post, though.)

I added a cifs mount and started scanning. Saw a surprising amount of memory pressure with 65M of swap. It does appear to be enough, though:

Every 1.0s: free -m                                                                   Wed Sep  7 13:28:25 2016

             total       used       free     shared    buffers     cached
Mem:           482        473          9         32         30        314
-/+ buffers/cache:        128        353
Swap:           63         40         23

(watch -n 1 free -m)

Node seems to be quite the memory hog but the kernel is able to keep some free memory and most of the used memory is for cache anyway. Or maybe Node has some memory management of its own? I really dunno.

Then I saw that a bunch of my albums were missing, and remembered that cifs doesn’t allow album directory names like “A Brief History of Rhyme: MC Hawking’s Greatest Hits” so I deleted all the user data, rebooted and configured it with nfs4.

This time around I actually measured things.

45 seconds to a login prompt over hdmi and at about 2:15 I got the startup chime and at about 2:30 the browser responded. This was repeated almost to the second with 128MB of swap! Let’s see more below:

The numbers from watch -n1 free -m while updating the library with 65MB of swap:

Every 1.0s: free -m                                                                   Wed Sep  7 13:58:08 2016

             total       used       free     shared    buffers     cached
Mem:           482        477          4         14          4        368
-/+ buffers/cache:        105        377
Swap:           63         52         11

Now that the library is scanned and I’m listening to music, those statistics are essentially unchanged.

And with 128MB and listening, which should be quite the same:

volumio@volumio:~$ free -m
             total       used       free     shared    buffers     cached
Mem:           482        475          6          4         69        247
-/+ buffers/cache:        158        323
Swap:          127          0        127

It would appear the kernel is more averse to swapping if there’s more of it!

So the next step was to go back to 65MB and set vm.swappiness = 40.

I did this by creating the file /etc/sysctl.d/10-swappiness.conf:

volumio@volumio:~$ cat /etc/sysctl.d/10-swappiness.conf 
vm.swappiness = 40

Statistics during play:

Every 1.0s: free -m                                                                   Wed Sep  7 14:41:33 2016

             total       used       free     shared    buffers     cached
Mem:           482        471         10          4         70        242
-/+ buffers/cache:        158        323
Swap:           63          0         63

During updating the library:

Every 1.0s: free -m                                                                   Wed Sep  7 15:03:10 2016

             total       used       free     shared    buffers     cached
Mem:           482        472         10         26          3        301
-/+ buffers/cache:        167        314
Swap:           63          2         61

Sure, I’m listening to music while I’m doing this, but it isn’t a perfect experiment yet. Having said that, I believe that a smaller swap size would have the following advantages:
[list=]If there’s something wrong with the overlay fs, it’ll actually fit on the file system![/list]
[list=]Because swap is the kernel’s last refuge, on slow i/o, for running out of memory, its use should be discouraged[/list]

There seems to be no disadvantage so far, as most of the memory goes to caches and even large flacs play well.

Looking at the dpkg -l list there’s very little I’d consider redundant :frowning: My favorite solution would be to go full appliance, like OpenELEC, but that’s quite a lot of effort and therefore infeasible.

For some reason the image I built myself shows the overlay filesystem as 1.6G big with only 78M used. This makes very little sense as only /usr/ reports 399MB for its size!

Why would it suddenly be different and report so little used space? Is there something in the flashing process that can affect the overlay filesystem’s size?

Would you consider dropping the swap file size and vm.swappiness?

Thanks a lot, love the new Volumio!

Very interesting indeed, and yes, I am willing to change the swappiness and swap size based on this findings. Some things to consider, however:

  • The most RAM intensive task is at the first boot, where we
    1- Copy plugins settings to /data/
    2- Resize images to /data/backgrounds

We added swap because the system was reporting out of memory error, but after this step there were no issue whatsoever.

So, to replicate this condition:

  • remove the content of /data/configuration and /data/backgrounds

  • kill node (this will restart volumio) killall node

Then, how the overlay works:

  • The system resides on a tar image, which gets loaded on RAM at boot time
  • We add on top of that an overlay partition, that will store the modification to the system
  • This means that if you edit\add\remove a file, the difference from the system tar image (squashfs image) will be written to the overlay partition allowing data persistence.
  • This way we’ll have both stability and 1:1 update with the squashfs image and ability to edit the files with overlay.

Hope this clarifies the system structure to you

let me know what you think

I tried this

volumio@volumio:~$ mkdir data
volumio@volumio:~$ mv /data/configuration/ /data/backgrounds/ data/
volumio@volumio:~$ sudo swapoff -a
[sudo] password for volumio: 
volumio@volumio:~$ free -m
             total       used       free     shared    buffers     cached
Mem:           482        466         15          4         70        244
-/+ buffers/cache:        150        331
Swap:            0          0          0

volumio@volumio:~$ sudo killall node && watch -n1 free -m

Every 1.0s: free -m                                                                   Thu Sep  8 14:28:30 2016

             total       used       free     shared    buffers     cached
Mem:           482        429         52          4         71        241
-/+ buffers/cache:        116        366
Swap:            0          0          0

Every 1.0s: free -m                                                                   Thu Sep  8 14:29:12 2016

             total       used       free     shared    buffers     cached
Mem:           482        477          4          4         70        126
-/+ buffers/cache:        280        201
Swap:            0          0          0

# at 14:29:35 I refreshed the browser, or tried to

Every 1.0s: free -m                                                                   Thu Sep  8 14:29:45 2016

             total       used       free     shared    buffers     cached
Mem:           482        478          3          4          0         14
-/+ buffers/cache:        463         18
Swap:            0          0          0

and it crashed completely.

After a reboot

volumio@volumio:~$ free -m
             total       used       free     shared    buffers     cached
Mem:           482        150        331          1         12         47
-/+ buffers/cache:         90        391
Swap:           63         59          4

So it apparently required quite a bit of RAM but recovered - as you can see, there’s a lot of free memory. Unfortunately it doesn’t find my NAS after this anymore.

Repeat experiment, this time going to check the Pi in the living room (it’s a bother to get it hooked up in my study)

volumio@volumio:~$ sudo swapoff -a
[sudo] password for volumio: 
volumio@volumio:~$ free -m
             total       used       free     shared    buffers     cached
Mem:           482        245        236          4         19         77
-/+ buffers/cache:        149        333
Swap:            0          0          0

volumio@volumio:~$ sudo rm -rf /data/configuration/ /data/backgrounds/ && watch -n1 free -m

Every 1.0s: free -m                                                                   Thu Sep  8 14:40:37 2016

             total       used       free     shared    buffers     cached
Mem:           482        214        267          4         21         84
-/+ buffers/cache:        109        373
Swap:            0          0          0

# I just missed the part where free went to like 4MB or somesuch, this dipped really low there

Every 1.0s: free -m                                                                   Thu Sep  8 14:42:07 2016

             total       used       free     shared    buffers     cached
Mem:           482        139        342          4          4         20
-/+ buffers/cache:        113        368
Swap:            0          0          0

# About a minute later try to refresh browser

Every 1.0s: free -m                                                                   Thu Sep  8 14:43:37 2016

             total       used       free     shared    buffers     cached
Mem:           482        478          3          4         33         67
-/+ buffers/cache:        378        104
Swap:            0          0          0

And the last reading I got was at 14:43:48 with the tv gone black.

I think it’s safe to conclude that some amount of swap is required :smiley:

After the boot

 volumio@volumio:~$ free -m
             total       used       free     shared    buffers     cached
Mem:           482        457         24          1          7         28
-/+ buffers/cache:        422         59
Swap:           63         30         33

# Time to restore the backup of /data/ and reboot one last time

volumio@volumio:~$ cp -a data/* /data/
volumio@volumio:~$ sudo reboot

volumio@volumio:~$ free -m
             total       used       free     shared    buffers     cached
Mem:           482        476          6          4         71        246
-/+ buffers/cache:        158        323
Swap:           63          0         63

The couple of experiments showed 59 and 30 megs of swap used, with something in caches.

If 59/64 is too close for comfort, it could be 80MB of swap, but it would be prudent to remember setting vm.swappiness = 40 regardless, because of the MicroSD situation. Swap size really did not increase performance earlier, quite the opposite, having swap on and requiring it during boot only slows it down.

I did one last experiment with 80MB, and this was 56M swapped after reboot with having deleted the stuff from /data/

volumio@volumio:~$ sudo swapoff -a
[sudo] password for volumio: 
volumio@volumio:~$ free -m
             total       used       free     shared    buffers     cached
Mem:           482        384         97          4         13         56
-/+ buffers/cache:        314        167
Swap:            0          0          0
volumio@volumio:~$ sudo swapon /data/swapfile 
volumio@volumio:~$ free -m
             total       used       free     shared    buffers     cached
Mem:           482        385         96          4         13         57
-/+ buffers/cache:        314        167
Swap:           79          0         79
volumio@volumio:~$ sudo killall node && watch -n1 free -m

Every 1.0s: free -m                                                                   Thu Sep  8 15:17:46 2016

             total       used       free     shared    buffers     cached
Mem:           482        288        193          4         29        114
-/+ buffers/cache:        145        336
Swap:           79          0         79

No worries with refreshing the browser.

volumio@volumio:~$ cp -a data/* /data/
volumio@volumio:~$ sudo reboot

# after reboot

volumio@volumio:~$ free -m
             total       used       free     shared    buffers     cached
Mem:           482        468         13          4         72        245
-/+ buffers/cache:        151        331
Swap:           79          0         79

# hit play in browser

volumio@volumio:~$ sudo killall node && watch -n1 free -m

# music keeps playing

Every 1.0s: free -m                                                                   Thu Sep  8 15:23:05 2016

             total       used       free     shared    buffers     cached
Mem:           482        445         36          4         72        247
-/+ buffers/cache:        125        356
Swap:           79          0         79

# after the browser loader thingie recovered

Every 1.0s: free -m                                                                   Thu Sep  8 15:24:24 2016

             total       used       free     shared    buffers     cached
Mem:           482        477          4          4         70        244
-/+ buffers/cache:        162        319
Swap:           79          0         79

So in light in all these findings, and because the Raspberry Pi is so slow and personally I can’t test this a whole lot more, I’d say that 80MB swap and vm.swappiness = 40 would leave some margin for error, or at least I couldn’t get it to break here.

Ok, sounds like a new version of union mounts, which is the last of this kind of technology that I’ve used personally for similar situations :slight_smile:

By the way if this makes it into a testing build, could you please ping this thread?

I have email notifications for this and would gladly give it a shot later if such a build comes out, but until then, I’m sticking to my custom 80M swap etc :slight_smile:

Thanks a lot!

Very interesting, and I think that 80 megs are the right amount, as you suggest. Of course, a little bit of testing won’t hurt before releasing to the public.
So I’ll apply those and make a test build soon!

So far, thanks!

Added github.com/volumio/Build/commit … abd758d48d