reindexing slow even with few new files - could be playlist issue.

Answered

Comments

20 comments

  • Official comment
    Avatar
    Tony W. - Product Support Manager

    Hi Mat

    Please select Help, Send Support Request so we may see the log file.

    This will definitely tell if it is a Rebuild Index or just an updated Reindex Collection.

    Comment actions Permalink
  • Avatar
    Mat

    An update... the count is now over 80000 in the reindexing
    it really looks like it is doing a complete rebuild rather than a fast reindex.
    i selected to reindex so this seems wrong

    0
    Comment actions Permalink
  • Avatar
    Mat

    i will try to do that.
    i can confirm that i am selecting reindex - not re build.
    i can report here that some of the lines in the log are:
    Dec  5 19:15:48 (none) user.info ./ms.pl: main::__ANON__ ./ms.pl (1233) No path match for imported playlist:
    followed by  /var/mnt/....
    and other lines are basically the same thing, except that it is reporting that a playlist has been imported.
    i can see this log by going to diagnostic and then clicking on more.

    0
    Comment actions Permalink
  • Avatar
    Mat

    also what i dont understand is why there is no reindex option on the webpage of the player.
    i.e. the page at 10.0.0.13 for the given player for example, does have a rebuild index but i cant see any reindex option.
    there is a reindex option in settings - media library in the ios app - but not on the webpage for the player that i can access in firefox.

    0
    Comment actions Permalink
  • Avatar
    Mat

    I am also seeing some lines like this:
    Dec  6 10:13:06 (none) user.info ./ms.pl: main::__ANON__ ./ms.pl (1233) imported playlist (!10CD10): /var/mnt/DEEPTHOUGHT-media/AudioBooks/Authors/.....

    0
    Comment actions Permalink
  • Avatar
    Tony W. - Product Support Manager

    Hi Mat

    This Help Centre Article may assist; https://support.bluesound.com/hc/en-us/articles/204103486

     

    0
    Comment actions Permalink
  • Avatar
    Mat

    Hi Tony.

    I have seen that article but I dont think it relates to my problem.  
    my players are not restarting when the index happens, nor does it seem to make a difference if I restart them.

    In fact, that article does illustrate what I did do origionally when some of the files in my library were not being indexed before the first library was created.  i did look at the log, identify the most recent files just before the error, i removed those files and re built - and the library was created.

    the problem I am having now is one related to the reindexing process going far too slow and the ios app reporting...

    indexed 3000 songs...
    indexed 3500 songs...
    indexed 4000 songs...
    ... an so on progressively up to...
    indexed 80500 songs etc.
    taking many hours to complete.

    restarting the players simply kills the reindexing process which then has to be started over.

    This is why i thought that it was doing a rebuild not a reindex.

    i can see in the log:
    Dec  6 09:51:50 (none) user.info ./ms.pl: main::HttpRequest ./ms.pl (908) [1] 10.0.0.10: /Reindex
    Dec  6 09:51:50 (none) user.info ./ms.pl: main::DoReIndex ./ms.pl (1188) Indexing collection LocalMusic: /var/mnt -> /var/data

    this does suggest reindexing rather than rebuilding...

    I have no idea what is happening and why it is so slow.

    0
    Comment actions Permalink
  • Avatar
    Tony W. - Product Support Manager

    Hi Mat

    80,000 should take about an hour and a half give or take. If it is taking significantly longer, we are trying to resize or resolve oversize artwork files. Please see this Help Center to see what may be happening. Article; https://support.bluesound.com/hc/en-us/articles/200271926

    If you want, you can also send us the log file by selecting Help, Send Support Request and one of our BluOS Support Crew members can assist.

     

    0
    Comment actions Permalink
  • Avatar
    Mat

    Thanks Tony.
    Do you mean it will take 1.5 hours to index 80000 songs. or that it will take 1.5 hours to go over an 80000 sized collection indexing only the few new files?
    i can say that my new index is up to 70500 files and it has been going for four or so hours.  this is a flex connected by ethernet to my router.
    as for artwork, i do not have any jpg files in the directories so the only artwork could be from the mp3 files themselves if there is embedded art, or from the internet if you are doing any database lookups.

    0
    Comment actions Permalink
  • Avatar
    Tony W. - Product Support Manager

    Hi Mat

    A good rule of thumb is about 1000 tracks a minute for an initial index or a Rebuild Index and about a 1/3 that time to Reindex a collection and add music.

    0
    Comment actions Permalink
  • Avatar
    Mat

    ok so i have built a new index and it took almost all night... so more than 7 hours to get to 150000 items
    on your calculation it should have taken 150 minutes - nearly 3 hours - give or take ... in my case it has taken more than double that time.
    in the case of reindex - if that is about 1/3 of the time of the build then a re index would take nearly 2.5 hours for me.
    it should be about 50 minutes on your calculations... approximatly.
    i know i have no jpg artwork - if there is artwork it is embedded ... i know there are lots of directories and m3u playlists.
    do you think the number of m3u files is making things slow?
    is there a way to reindex but to ignore m3u files...

    0
    Comment actions Permalink
  • Avatar
    Mat

    I dont mind build taking a long time, but reindex should be very snappy so that new files can be found quickly and slotted into the library.
    my expectation is that a reindex should take 5 - 10 minutes or so if there are not many additional files to find.

    0
    Comment actions Permalink
  • Avatar
    Tony W. - Product Support Manager

    Based on the size of your library, a Reindex should take 15-20 Minutes. Send us the log file by selecting, Help Send Support request and we can verify if M3U files are slowing it down as well. You cannot disable Playlist importing as the vast majority of consumers demand it.

    0
    Comment actions Permalink
  • Avatar
    Mat

    Re the idea of not importing playlists... I think I just meant to have an option to turn that off.  users who want it would not turn it off.

    maybe what I am suggesting is some kind of ignore list - ie. a list of file extentions that will be ignored by the indexer.  that way, one could include *.m3u - or any other file type - in that ignore list.  by default the list would be empty. perhaps this is a broader feature request idea.


    but in any case, the m3u files may not even be the reason for my slow reindexing.  it could be something else.

    as an aside, if i do go to menu - my playlists in the blue os app, i end up seeing nothing in the list and the player seems to reboot.  it is as if the player is crashing on something when i try to access the playlists that it has imported.  maybe there are too many fot it?

    0
    Comment actions Permalink
  • Avatar
    Tony W. - Product Support Manager

    Hi Mat

    If you are continuing to have issues, please select Help, Send Support Request so we may see your log file.

     

    Thanks

    0
    Comment actions Permalink
  • Avatar
    Mat

    I think i have solved the slow indexing problem.  i think the large number of m3u playlists was making it go far too slow.
    i deleted all the m3u files from the share and built a new index.
    the log reports that 100000 files took about 2.5 hours ... baring in mind that this was done over wifi rather than being plugged into ethernet
    so while still a bit slow... i think this is sort of within range of Tony's rule of thumb.
    once the index is complete i will try a reindex to see how long that takes but i do anticipate a much faster reindex than before.

    as for artwork... i am wondering why the bluesound needs to resize it during the build of index process... rather than leaving it as is (which could be quicker) and letting the ios app do the resizing when it is displayed.  iphones and ipads will need to resize anyway given the different screen sizes
    maybe it is because the bluesound players have a limit to their storage such that artwork has to be resized to fit for large libraries.
     artwork files can take up a lot of this space.



    1
    Comment actions Permalink
  • Avatar
    Tony W. - Product Support Manager

    Hi Mat

    Great news!

    Thanks for the feedback! Unfortunately there is far more processing power in the Bluesound Player. Doing this in the handheld app would slow it down even more - especially on lower grade Android devices and older iPhones. 

    0
    Comment actions Permalink
  • Avatar
    Mat

    well reindex confirms it... the large number of m3u playlists seemed to be causing the slow inde and reindex
    after removing al the playlists and starting over...
    the 150000 files took 4 hours to index from scratch over the ethernet connection.  this is a bit slow but i can handle that
    a reindex only took 400 seconds with a handful of new files added.  so that means less than 7 minutes to reindex a library of 150000 files.
    that seems far more acceptable to me.

    I am still getting some errors of the kind: buffer_get_int_le: buffer error at /usr/l
    but these do not seem to halt the indexing process like the other user.error that i had some days ago.

    So i wonder how are m3u playlists handled?
    it seems that, unlike files that have not changed, an m3u playlist is truely re-done on every reindex even if it were there before.
    when a playlist is encountered on reindex, does the bluesound re-index every file that the m3u refers to even if they were there previously?

    my only suggestion here is to ensure that playlists are not 're-done' if the same ones are encountered again on a reindex.
    it's ok to index them on first go, as i am sure people want the players to recognise playlists... but re-doing them on reindex is what can caused a reindexing process to be as slow as a first build, which I am sure is not what people want.

    0
    Comment actions Permalink
  • Avatar
    Mat

    oh an on the issue of artwork, i was not suggesting that the iphone would do the resizing of the artwork for the library.
    i think my suggestion was to not bother resizing at all in the library.  this would save time.
    then only when the iphone goes to display the artwork, it will automatically be resized just like how images are resised to be shown in a browser like safari.

    1
    Comment actions Permalink
  • Avatar
    Tony W. - Product Support Manager

    Glad to hear Mat - Artwork resizing was just added to resolve issues with artwork not being displayed at all or taking a long time to load in the App.

    0
    Comment actions Permalink

Please sign in to leave a comment.