You are here

Forum Topic Pruning: SOS needs your help

All about the tools and techniques involved in capturing sound, in the studio or on location.

Forum Topic Pruning: SOS needs your help

Postby Forum Admin » Sat May 02, 2020 8:11 am

Hi everyone,

Our site is being negatively affected by the massive size of our database, and a vast amount of server processing and resources/caching are having to be devoted to the (phpBB) SOS Forum topics/posts. This is impacting on page-load times all across the rest of the site and our search engine's ability to perform optimally without timing out.

The SOS Forum currently has 616,472 Posts and 57,317 Topics. Regrettably, we need to reduce its Post Count by approximately one-third, circa 200,000. Post-purge, we will also need to implement phpBB's regular Post-pruning routines, which can be configured individually per sub-forum, so the post quantity does not grow massively and the oldest topics/posts will get constantly erased.

We need to action this initial pruning as soon as possible and propose to cull the oldest Topics/Posts on one or two forums at a time to reduce the massive server overhead pruning causes. We anticipate the pruning and our search engine re-indexing the remaining content taking 2 weeks to complete.

From our checks, we will need to erase every Topic (and its Posts) that has not been posted to since 01 January 2012. Achieving this will reduce the overall Post Count by the approximated 200,000 figure.

Here's where SOS needs your help...

Topics from pre-2012 that you value and see as a benefit to this community can be retained.

However, we need your help to identify them. So please look through your saved Bookmarked Topics or jump to the oldest pages of your favourite sub-forum(s) and skim through the Topic list.

When on a Topic you wish us to keep, please click on the REPORT THIS POST icon (an exclamation mark inside a circle) and type something like 'Do Not Prune' in the message box and submit it.

One of the Moderators or myself will then move that Topic and all its Posts into the existing USEFUL INFORMATION ARCHIVE (UIA) sub-forum and the Topic will become a read-only item. It will be searchable using the Forum/Main Search functions. If you then wished to 'reply' to the read-only Topic, you'd need to start a new Topic, link to the archived one, and continue discussions in the new Topic. Not ideal, but workable.

When can we start and how much time do we have?
You can start reporting valued pre-2012 Topics from today and we'll need to stop taking requests by 7pm UK time on Friday 15 May 2020.

Thanks for your understanding.

cheers,
Ian G
User avatar
Forum Admin
Moderator
Posts: 3402
Joined: Mon Jul 21, 2003 11:00 pm
Location: A studio deep in the fenlands of Cambridgeshire, UK
SEARCH 11,508 SOS Reviews/Techniques articles: www.soundonsound.com/search

Re: Forum Topic Pruning: SOS needs your help

Postby desmond » Sat May 02, 2020 8:51 am

Is there any way to export a pre-pruned text dump of the forums, which can then be hosted online?

It's a shame in the age of big data that online history might be deleted, but I understand the performance and business needs to do this. But if some kind of text archive could be exported and hosted, then at least the data would be retained and be searchable just in case, rather than have a bunch of text gone forever.

Admittedly, how much value is in those posts is probably questionable, but even so...

Or is it all just a big conspiracy to stop me reaching my 10K post count..? (Again!) :lol:
User avatar
desmond
Jedi Poster
Posts: 10243
Joined: Tue Jan 10, 2006 1:00 am
mu:zines | music magazine archive | difficultAudio

Re: Forum Topic Pruning: SOS needs your help

Postby Forum Admin » Sat May 02, 2020 9:45 am

Hi Desmond,

We plan to take a full database backup prior to pruning and store it offline.

As to a text-dump, I've been told it's not a simple task since phpBB pulls in who posted, etc from our CMS user base, and it'll require our web team to spend expensive time to achieve this. That's time we don't have at present with all our other pressing projects and we urgently need to lighten the server load or we may find ourselves with patches of 'dark time', temporary Forum suspension, or any other undesirable scenario -- and commercially, that simply can't happen.

So the proposed solution is the optimal one.
User avatar
Forum Admin
Moderator
Posts: 3402
Joined: Mon Jul 21, 2003 11:00 pm
Location: A studio deep in the fenlands of Cambridgeshire, UK
SEARCH 11,508 SOS Reviews/Techniques articles: www.soundonsound.com/search

Re: Forum Topic Pruning: SOS needs your help

Postby desmond » Sat May 02, 2020 10:12 am

Understandable - thanks Ian.

How are important/saved threads immune to auto-pruning in the future?
User avatar
desmond
Jedi Poster
Posts: 10243
Joined: Tue Jan 10, 2006 1:00 am
mu:zines | music magazine archive | difficultAudio

Re: Forum Topic Pruning: SOS needs your help

Postby Hugh Robjohns » Sat May 02, 2020 11:08 am

The UIA forum will be the only forum excluded from the pruning regimen.
User avatar
Hugh Robjohns
Moderator
Posts: 28570
Joined: Fri Jul 25, 2003 12:00 am
Location: Worcestershire, UK
Technical Editor, Sound On Sound

Re: Forum Topic Pruning: SOS needs your help

Postby Forum Admin » Sat May 02, 2020 11:25 am

I think desmond is referring to the ongoing auto-pruning of sub-forums after the initial prune has occurred.

Valid question. Each forum will have settings for its Topics and only Topics that have not been posted to in NNNN days would be candidates. And we still have to decide on each setting for the post-purge era. Unfortunately, we can't use a parameter like Days since last viewed, because there are so many search engine bots coming to the SOS Forum and crawling old (and new) posts on a daily basis.
User avatar
Forum Admin
Moderator
Posts: 3402
Joined: Mon Jul 21, 2003 11:00 pm
Location: A studio deep in the fenlands of Cambridgeshire, UK
SEARCH 11,508 SOS Reviews/Techniques articles: www.soundonsound.com/search

Re: Forum Topic Pruning: SOS needs your help

Postby desmond » Sat May 02, 2020 12:07 pm

Sorry - yes, Ian is correct - to be more clear, let's say there was a thread from 2013 that contained content that some felt was important to be preserved. At the moment, we'd alert a mod and say "don't delete this" as part of the general prune.

But later on, auto-prune would be deleting old threads without mod intervention. So even though people had informed mods that the thread was important and shouldn't be deleted, once it fitted the auto-prune criteria in regards to date range / last posted etc (whatever that ends up being), then it will presumably be deleted automatically.

I wondered if there was going to be some mechanic or flag on threads that could be used so auto-prune wouldn't automatically delete some old but gold content...
User avatar
desmond
Jedi Poster
Posts: 10243
Joined: Tue Jan 10, 2006 1:00 am
mu:zines | music magazine archive | difficultAudio

Re: Forum Topic Pruning: SOS needs your help

Postby Hugh Robjohns » Sat May 02, 2020 12:14 pm

I think the plan is to ~move~ any threads or individual posts that have been deemed valuable across into the Useful Information Archive.

As I understand it, the alternative way to prevent auto-deletion is to mark posts as 'Stickies', but as they then inherently appear at the top of the first page in each forum it wouldn't take that many saved threads before there'd be nothing new on the first page, which is clearly not a good thing!
User avatar
Hugh Robjohns
Moderator
Posts: 28570
Joined: Fri Jul 25, 2003 12:00 am
Location: Worcestershire, UK
Technical Editor, Sound On Sound

Re: Forum Topic Pruning: SOS needs your help

Postby Arpangel » Sat May 02, 2020 12:20 pm

No problems here, anything to keep the forum going.
User avatar
Arpangel
Jedi Poster
Posts: 5285
Joined: Sat Jul 12, 2003 12:00 am

Re: Forum Topic Pruning: SOS needs your help

Postby desmond » Sat May 02, 2020 12:21 pm

Hugh Robjohns wrote:I think the plan is to ~move~ any threads or individual posts that have been deemed valuable across into the Useful Information Archive.

Ok, so they'd come out of their old sub-forum home to the new archive place. Would external links pointing directly to those threads continue to work, or would they break?

Hugh Robjohns wrote:As I understand it, the alternative way to prevent auto-deletion is to mark posts as 'Stickies', but as they then inherently appear at the top of the first page in each forum it wouldn't take that many saved threads before there'd be nothing new on the first page, which is clearly not a good thing!

Yes, I don't think that's a workable option, really...
User avatar
desmond
Jedi Poster
Posts: 10243
Joined: Tue Jan 10, 2006 1:00 am
mu:zines | music magazine archive | difficultAudio

Re: Forum Topic Pruning: SOS needs your help

Postby Forum Admin » Sat May 02, 2020 12:31 pm

desmond wrote:Ok, so they'd come out of their old sub-forum home to the new archive place. Would external links pointing directly to those threads continue to work, or would they break?

I'm pretty certain phpBB forum software creates an automatic redirect to its new home in the UIA forum, and then search engines would re-index it.

I'll do some tests over the weekend to confirm or deny this.
User avatar
Forum Admin
Moderator
Posts: 3402
Joined: Mon Jul 21, 2003 11:00 pm
Location: A studio deep in the fenlands of Cambridgeshire, UK
SEARCH 11,508 SOS Reviews/Techniques articles: www.soundonsound.com/search

Re: Forum Topic Pruning: SOS needs your help

Postby Forum Admin » Sat May 02, 2020 12:45 pm

OK, my test is 100% conclusive...

Original URL in SOS Support Forum:

https://www.soundonsound.com/forum/view ... 30&t=69575

Topic moved to UIA forum (using Quick Mod Tools at the foot of the Topic) where the URL becomes:

https://www.soundonsound.com/forum/view ... 30&t=69575

As you can see, they are identical URLs. To convince myself I closed my browser (Safari), opened Chrome and pasted in the original URL. Naturally, it took me directly to the Topic and its three posts, all of which now reside in the Useful Information Archive forum.

Result. :thumbup:
User avatar
Forum Admin
Moderator
Posts: 3402
Joined: Mon Jul 21, 2003 11:00 pm
Location: A studio deep in the fenlands of Cambridgeshire, UK
SEARCH 11,508 SOS Reviews/Techniques articles: www.soundonsound.com/search

Re: Forum Topic Pruning: SOS needs your help

Postby desmond » Sat May 02, 2020 1:03 pm

Great! :thumbup:
User avatar
desmond
Jedi Poster
Posts: 10243
Joined: Tue Jan 10, 2006 1:00 am
mu:zines | music magazine archive | difficultAudio

Re: Forum Topic Pruning: SOS needs your help

Postby ConcertinaChap » Sat May 02, 2020 2:56 pm

Does that mean our post counts will all be reduced and we'll all lose our Jedi status and be busted back to the ranks?

:cry:

CC
User avatar
ConcertinaChap
Jedi Poster
Posts: 9110
Joined: Wed Jul 20, 2005 12:00 am
Location: Bradford on Avon
Making music: Eagle Alley, recording music: Mr Punch's Studio
A full wine rack is a happy wine rack.
 

Re: Forum Topic Pruning: SOS needs your help

Postby MOF » Sat May 02, 2020 2:58 pm

Everything I’ve written on this forum is pure gold and should be preserved for the nation. :lol:
Does this technical change affect my chance of ever becoming a Jedi poster?
MOF
Frequent Poster
Posts: 1232
Joined: Thu Mar 06, 2003 1:00 am
Location: United Kingdom

Next