Music Import via cron


#1

I am trying to build a work flow that permits executing the import script via a cron job.

I get this to work, however it looks like it duplicates the tracks. If i go to the library in the GUI, the number of tracks and reported space increments even though there is not any net new files.

I am using --inplace --async, and --replace flags.

0.17 - i have not attempted to upgrade to 0.18 as thetarkus’s docker image has not been updated yet.

–izz


#2

Hi @izz

We do have a 0.18 version for the all-in-one image. Simply use funkwhale/all-in-one:0.18 as the image name, we build this image for every release, using https://github.com/thetarkus/docker-funkwhale :slight_smile:

Would you mind sharing the cronjob script you’re using? I’d like to reproduce that :slight_smile:


#3

edit RTFM error, resolved.


#4

I was able to get the new install built. (RTFM).

funkwhale:
image: funkwhale/all-in-one:0.18
container_name: funkwhale
environment:
- PUID=1024
- PGID=1000
- FUNKWHALE_HOSTNAME=my.private.host:3030
- FUNKWHALE_PROTOCOL=http
- DJANGO_SECRET_KEY=bf77ba1f0b55066fa32f7670030dasdfasdfad4ca0ebd213bb8ffc33ba51a1
volumes:
- /etc/localtime:/etc/localtime:ro
- /mnt/siren/Media/Music:/music:ro
- /mnt/siren/Media/FunkWhale:/data

image

[email protected]:/Music$ find . -type f -name "*.flac" | wc -l
12392

[email protected]:~$ docker exec -it funkwhale manage import_files $LIBRARY_ID "/music/**/**/*.flac" --verbosity 0 --in-place --async --replace --recursive --noinput
Successfully launched import for 7304 tracks

Ok, this will take a bit to import. I will respond again once this one is done.
I am curious why I am only able to see 7304 tracks, when as shown in the find, there is over 12k.

Being new user, can only put one image, in post. The GUI shows 7304 tracks.

–izz


#5

@izz you should not share your DJANGO_SECRET_KEY anywhere, it should be considered as a password. Please regenerate one on your server if it was your actual secret key :slight_smile:

I think it’s because of this pattern: "/music/**/**/*.flac". "/music/**/*.flac" should yield more results.

Did the import work ok otherwise?


#6

The import does work,I have 7304 tracks.

Ok, I swore I tried that directory depth the first time I did the import, it did pick up all 12392 tracks this time.

[email protected]:~$ docker exec -it funkwhale manage import_files $LIBRARY_ID "/music/**/*.flac" --verbosity 0 --in-place --async --replace --recursive --noinput
Successfully launched import for 12392 tracks

Did does show you what I was raising the first case over though.

image

Notice, processing 12392 which is the right amount, but the number of tracks listed in the library is additive from the first add.

I am uncertain if this is cosmetic using the cumulative? Or pulled from postgres directly. This value never changes even after the full import is done. So if I run that same command again, with the --replace flag, it will increment by an additional 12k tracks.

–izz


#7

We probably have an issue with the replace flag then. Do you mind deleting the library and relaunching an import without the flag? Also, if you’re running the task via a cronjob, it’s probably better not to use that flag, because it will only work on newly add/not imported files this way :slight_smile:


#8

Is there any way to target the postgress sql directly and do an alternate load? I can write a script to import.

I am not a fan of doing a delete then an add, considering that the add takes 90 minutes and I do not want to impose that on my family. My library is around 20k total tracks, I was only importing the flac in that import.

May I request a feature enhancement to provide a mechanism to do scheduled import that can handle duplicates?

–izz


#9

Probably, but I would not recommend that, because there is a lot of logic and edge cases to handle.

I was suggesting the deletion as a way get rid of the additionnal tracks issue until it is fixed. You would not have to delete everytime you run an import. If the first import takes 90 minutes, and you only have a few new tracks next time you run the cronjob, it should complete in seconds. This is, of course, assuming you’re not using the --replace flag

I’m not sure what you mean by that. If you try to import two files that are really the same track, Funkwhale should import the first and skip the second.

However, you are facing a bug with the --replace flag, and I suggest you don’t use the flag until it is solved :slight_smile:


#10

Here is the rub, I have another music manager that manages my music, this is why I am using the in-place to begin with.

The music manager will update the music as I get it, and then I want funkwhale to service it up to my family. So the cron will run against my library, not a migratory process outside of my existing workflow.

–izz


#11

I understand that @izz, I’m simply suggesting not using the --replace flag. You can keep the --in-place one :slight_smile:


#12

If I simply do the --in-place it duplicates the tracks. I tried the --replace to eliminate the duplication.

I will see what alternatives I have.


#13

I’ll try to reproduce your issue, sorry for the trouble!