Backup of the original Steam Forum

Subject: Backup of the original Steam Forum Tue 11 Jul 2017 - 20:53

Toony and I are attempting to backup the old 1700+ page Steam forum, and will be uploading each page to this post as we go. So without further ado... Lets see if we can make this work!

Ok, so I'm running into a big issue. The first page of the forum alone ends up being a 70 page pdf... much too long to copy into spoilers like below. In order to preserve google searchability I need to find a better way to do this.

Steam Forum Page 1, Part 1:

Subject: Re: Backup of the original Steam Forum Wed 12 Jul 2017 - 10:14

da_raabi wrote:: Toony and I are attempting to backup the old 1700+ page Steam forum, and will be uploading each page to this post as we go. So without further ado... Lets see if we can make this work!

Ok, so I'm running into a big issue. The first page of the forum alone ends up being a 70 page pdf... much too long to copy into spoilers like below. In order to preserve google searchability I need to find a better way to do this

In what form can you access the original, please? Is there any way you can store it as simple text so it can be compiled it as one continuous searchable document?

The text you've supplied here is a 726 line, 30Kb text file, easily searchable by Notepad++, or Google once it's been made public, maybe stored on TB's website

If you're happy to send it to the forum, or even to me personally, I'd be happy to help compile it

Subject: Re: Backup of the original Steam Forum Wed 12 Jul 2017 - 10:22

da_raabi wrote:: Toony and I are attempting to backup the old 1700+ page Steam forum ... In order to preserve google searchability I need to find a better way to do this.

I thought all that valuable info was lost, it is very encouraging to hear that we may be able to make it available again. Well done!

As this is a plain-text archive, perhaps it could be copied into a series of simple HTML web pages. Web pages would be directly browseable online, and would be open to search engines. It would also be relatively easy (if time-consuming) to add hyperlinks or other navigational aids later on if required. Perhaps TB could host it on his website.

Subject: Re: Backup of the original Steam Forum Wed 12 Jul 2017 - 13:56

Now we are talking. Toony has converted the first 50 pages of the forum, although 9 of those pages were missing/lost forever. He is the friggen champ!

Currently each page is stored as a pdf. The earlier pages in the forum convert to plain text fairly well, however many of the later pages do not. It's because Bing stored cache files of the earlier pages in a more plain text, while the later pages are stored in their original form (with pics, signatures, forum buttons etc) which makes converting to plain text downright disgusting.

I like the idea of converting to simple html web pages. I'm trying to make sure that all this stuff remains searchable via google, as many folks (who may not be a part of any group or forum) will likely use google to get the info they need. I was one of them until recently.

Here are two pages as Toony has sent them to me in my dropbox. This is what the earlier pages look like right now.

[You must be registered and logged in to see this link.]

Subject: Re: Backup of the original Steam Forum Wed 12 Jul 2017 - 13:59

I know a little html, so I'm going to attempt to put the first page into a simple low-format plain-text html page.

If it works, I like the idea of hosting everything on TB's page and linking to it from here.

Keep in mind... Toony has converted the first 50 pages. Thats amazing for one evening's work. BUT there are 1700 pages in that forum. This could take a while!

That is why I am bitching up a storm with Steam right now to see if I can get the old forum back up again. If that were to happen I think this process should still finish in case Steam pulls this again (not like this is the first time anyway).

Subject: Re: Backup of the original Steam Forum Wed 12 Jul 2017 - 14:35

da_raabi wrote:: I know a little html, so I'm going to attempt to put the first page into a simple low-format plain-text html page.

If it works, I like the idea of hosting everything on TB's page and linking to it from here.

Keep in mind... Toony has converted the first 50 pages. Thats amazing for one evening's work. BUT there are 1700 pages in that forum. This could take a while!

That is why I am bitching up a storm with Steam right now to see if I can get the old forum back up again. If that were to happen I think this process should still finish in case Steam pulls this again (not like this is the first time anyway).

I have some experience with creating web pages if you need help with that. Ditto if there is anything I can do to help with the Bing pages. NB: if several people are working on the same task, someone would need to be in control so we know who is doing what.

It would be helpful if Steam would put the pages back, even temporarily. I'm not sure that I would trust them long term, though.

Subject: Re: Backup of the original Steam Forum Wed 12 Jul 2017 - 14:35

da_raabi wrote:: Here are two pages as Toony has sent them to me in my dropbox. This is what the earlier pages look like right now.

[You must be registered and logged in to see this link.]

You're right in that these pages are a textual mess, but it's not impossible...

I'm starting to compile them, but page 1 starts in 2012, while page 2 starts in 2011. Any reason for this, please?

Subject: Re: Backup of the original Steam Forum Wed 12 Jul 2017 - 17:21

Sword2012 wrote:

da_raabi wrote:: Here are two pages as Toony has sent them to me in my dropbox. This is what the earlier pages look like right now.

[You must be registered and logged in to see this link.]

You're right in that these pages are a textual mess, but it's not impossible...

I'm starting to compile them, but page 1 starts in 2012, while page 2 starts in 2011. Any reason for this, please?

Its funny, Toony had said the same thing, but I'm not seeing that.

Page 1, post 1 is by TB on 08-01-2011, 09:50PM
Page 1, last post is by motanut on 09-30-2011, 10:21 AM
Page 2, post 1 is by KarlMonster on 09-30-2011, 12:08PM
etc

Toony sent me the first 50 pages he has found, and it seems that the ones found on Bing (like what I uploaded above) are synchronous, but those that were found through Google are not. For instance:

[You must be registered and logged in to see this link.]

See the problem? I honestly don't know quite why this is occuring. And why are bing pages 70 pages long in the pdf while each google page is only 3-4?

Subject: Re: Backup of the original Steam Forum Wed 12 Jul 2017 - 17:25

Strike that. I think I figured it out.

Bing is apparently pulling in dozens of pages when it says only page 1. For instance. If you look at [You must be registered and logged in to see this link.] then compare it to [You must be registered and logged in to see this link.] you can see that the first post from the Google Page 27 is on page 46 of the Bing PDF.

For whatever reason Bing is combining MASSIVE numbers of pages onto one page. This will take some weeding to clean up I think. It looks like Google can be skipped for this process though.

Boy this is confusing.

Subject: Re: Backup of the original Steam Forum Wed 12 Jul 2017 - 21:26

Please try this on bing.com:

"stalkersoup page 63 site:forums.steampowered.com"

Then you get 2 results for page 63 (the first and second result). The first is marked with [Archive].
They are completely differrent (see date of postings and length of page).

Is it possible, that the steam forum was "shortened", changed page length or started new at one time point - or perhaps bing changed mode of caching at some time point? I don´t know.

I will continue archieving (with CutePDF as you said) what i can find on bing and name the files Page0001.pdf ... Page0111.pdf... and so on.
When i can find a page only on google, then Page0112_google.pdf...
When i get 2 results on bing then Page0063_arch.pdf and Page0063, because the newer pages (try page 1611, 1712) look like the result not marked with [Archive], and i think, that they are the "right" ones. (They are shorter, but i think in the right order)
Missing pages i will write in a text file. I will do this in steps of 50 pages and send them to you.
If anyone wants to help, should do it the same way and start from 250, 500, 750, 1000, ... in blocks of 50?

Please post, if you or someone else think, that i´m wrong with something or knows a better way.

@da_raabi: posted this instead of email, because then others can help, too

Subject: Re: Backup of the original Steam Forum Wed 12 Jul 2017 - 21:38

For page 1228 the results from google and bing (not marked with [Archive]) seem to be identical, so i think the [Archive]-marked results are somehow "wrong"

Subject: Re: Backup of the original Steam Forum Wed 12 Jul 2017 - 22:08

Searching for page 90 with GOOGLE tells me 2 different results, one marked with [Archive] and one not.

When i look at the dates of posting, i think, that ALL results marked [Archive] wheter on bing or google are useless/kind out of right order and we should only save the ones without the [Archive]-marker.
The problem is, that the lower numbers of pages are even not available on bing.

Subject: Re: Backup of the original Steam Forum Wed 12 Jul 2017 - 22:20

(Sorry for spamming multiple posts)

So i made a mistake with my file names, because i forgot the "_arch" in the filenames, when i only found ONE result on bing (wrong thougts in post 21.26h).

The first 100 pages i made --> garbage Mad

Any ideas?

Subject: Re: Backup of the original Steam Forum Wed 12 Jul 2017 - 22:46

If the archives are that confusing, it might be easier to copy everything into temp text files and sort out later. If you create a complete set of empty PDFs, each one covering a block of dates (e.g. one PDF for every 3 months?), you (we) could then copy the temp text files into the relevant PDF by article date. Duplicates would be removed as you work through each file. Any additional volunteers would work on specific PDF(s) by date.

**Experienced** Posts : 156 Join date : 2017-06-02

One of the problems here is that we need the text to be easily searchable.
Remember, many of the links will be long gone and dead, so the only real purpose of the archives is fixes for broken missions and storyline related help.

I would even go as far a to propose that the files be kept in .txt format or, something that can be opened in word, or wordpad. Let me know what you guys think

However... OK how many accurate pages do ou have so far?

Subject: Re: Backup of the original Steam Forum Thu 13 Jul 2017 - 0:50

Steve90k wrote:: I would even go as far a to propose that the files be kept in .txt format or, something that can be opened in word, or wordpad. Let me know what you guys think

I'm for text every time. So much more flexible, searchable, and incorruptible...

Not even Word or PDF, just plain text as seen in Notepad

Subject: Re: Backup of the original Steam Forum Thu 13 Jul 2017 - 9:53

Sword2012 wrote:

Steve90k wrote:: I would even go as far a to propose that the files be kept in .txt format or, something that can be opened in word, or wordpad. Let me know what you guys think

I'm for text every time. So much more flexible, searchable, and incorruptible...

Not even Word or PDF, just plain text as seen in Notepad

PDF was developed as a final document sharing or printing format, not as an intermediate editing format. I agree that it would be better to stick with text where possible. However, there is formatting in the original web pages that I assume we would want to keep, for example paragraphs, line spaces and divisions between posts. Without that, you may end up with one continuous and unreadable block of text.

Unfortunately I can't get to the archived pages to test this, as when I try the Bing links they are all redirected to [You must be registered and logged in to see this link.] Toony, can you tell me how you reach the archived pages?

Subject: Re: Backup of the original Steam Forum Thu 13 Jul 2017 - 10:18

"stalkersoup page 63 site:forums.steampowered.com" in bing. Next to results is a triangle. Click on it and then click on "in cache"

Subject: Re: Backup of the original Steam Forum Thu 13 Jul 2017 - 11:36

Thanks Toony.

Copying an archived page into Notepad or Notepad++ does retain paragraph formatting. See
[You must be registered and logged in to see this link.]

Copying directly into a blank HTML page in Dreamweaver gives a better result and retains the horizontal rules (lines) between posts - see example at
[You must be registered and logged in to see this link.]

However, unless you are happy using a HTML editor, plain text would probably be easier to work with.

Subject: Re: Backup of the original Steam Forum Thu 13 Jul 2017 - 13:45

pchester wrote:: Copying an archived page into Notepad or Notepad++ does retain paragraph formatting

May I suggest that using Notepad++ is immeasurably more intelligent and intuitive than Notepad...

Subject: Re: Backup of the original Steam Forum Thu 13 Jul 2017 - 14:06

Well Toony had sent me quite a few pages that he downloaded as PDFs. Here is a combined PDF encapsulating everything he could find from 2011-2012. This is only through page 31 of the 100 he sent me so far...

[You must be registered and logged in to see this link.]

I do like how the dreamweaver page turned out though. I have an ancient copy of DW, how did you get the text into that format?

**Experienced** Posts : 156 Join date : 2017-06-02

Word 2010 can open PDFs and save them as text files, but generally a direct copy and paste of the website text can be used to paste it directly into a notepad file.

Subject: Re: Backup of the original Steam Forum Thu 13 Jul 2017 - 14:25

da_raabi wrote:: I do like how the dreamweaver page turned out though. I have an ancient copy of DW, how did you get the text into that format?

Very simply ... in the original archive web page, place cursor at top of message text, scroll to bottom and shift-click to select everything in between. Ctrl-C to copy selected text. In Dreamweaver, open a new (blank) HTML page, place cursor in the page in Design view, Ctrl-V to paste. No additional formatting was done. The .txt file was done the same way using Notepad++.

The HTML page(s) could be based on a template if we wanted them to match those on the hosting website.

I don't think it matters which editing method each person prefers to use, as long as the end result is the same.

Subject: Re: Backup of the original Steam Forum Thu 13 Jul 2017 - 16:22

pchester wrote:

da_raabi wrote:: I do like how the dreamweaver page turned out though. I have an ancient copy of DW, how did you get the text into that format?

Very simply ... in the original archive web page, place cursor at top of message text, scroll to bottom and shift-click to select everything in between. Ctrl-C to copy selected text. In Dreamweaver, open a new (blank) HTML page, place cursor in the page in Design view, Ctrl-V to paste. No additional formatting was done. The .txt file was done the same way using Notepad++.

The HTML page(s) could be based on a template if we wanted them to match those on the hosting website.

I don't think it matters which editing method each person prefers to use, as long as the end result is the same.

Ok, that works for the older pages that had already been de-formatted, however it does not work well for newer pages like this:

[You must be registered and logged in to see this link.]

Direct copy-paste comes out... ugly

**Experienced** Posts : 156 Join date : 2017-06-02

da_raabi wrote:

pchester wrote:

da_raabi wrote:: Direct copy-paste comes out... ugly

Seems not too bad for me. In this page there are only 2 pieces of information that are remotley usefull..

1.

uncalibrated psy helmets.
hey guys,

can anyone remember the name for removing the uncalibrated psy helmets using esc+f5 from my inventory?

i found one on the dead military guy in the forgotten forest somewhere, and i don't remember where i got the other one. but it is 2kg i don't need to be carrying!

2.To anyone who used the temporary workaround to get out of enforced God Mode - don't forget to remove the old 'inventory_sect' file, and 'creatures' folder from your gamedata/config folder, otherwise the game will CTD before it even starts. Trust me on this...
Last edited by sword2012: 02-28-2015 at 06:30 PM.
sword2012 is offline Reply With Quote
Everything else is either useless or outdated.

I uploaded a copy of the text file here...

[You must be registered and logged in to see this link.]

The text shows up exactly the same in notepad.

The thing is, almost 90% of this information will be useless once the faults are ironed out of the game. I dont know how long that will take, or even if the faults are being worked on right now, but as far as asthetics are concerned, I personally wouldn't give the asthetics any prioirty at all, providing the text is searchable..

Personally I would cut the text down to this...or even more!!!

[You must be registered and logged in to see this link.]

Subject: Re: Backup of the original Steam Forum Thu 13 Jul 2017 - 18:52

Steve90k wrote:

da_raabi wrote:

pchester wrote:

da_raabi wrote:: Direct copy-paste comes out... ugly

Seems not too bad for me.

Agreed, that is messy to copy direct to HTML, because every post is held as a separate set of nested tables and DIVs. Copied via plain text it looks more acceptable.

This is a direct copy from the new format archive page into a blank HTML page in Dreamweaver
[You must be registered and logged in to see this link.]

This is a direct copy as plain text in Notepad++ (pretty much the same formatting as Steve90k's example)
[You must be registered and logged in to see this link.]

And this is the notepad++ text file copied again into a blank web page
[You must be registered and logged in to see this link.]

The Notepad++ route seems the best answer here.

There are 1700+? pages to work through. It would be nice to cut them down to show only current/useful info, but it would be quite a job!

Subject: Re: Backup of the original Steam Forum Thu 13 Jul 2017 - 19:07

pchester wrote:: The Notepad++ route seems the best answer here.

There are 1700+? pages to work through. It would be nice to cut them down to show only current/useful info, but it would be quite a job!

If you do the hard work of getting it all into Notepad++ first, which is by far the most important stage, we can then work out a preferred layout before offering out the workload to share

We may even find someone good at scripting, who can separate the articles into the correct date order, and removing superfluous chunks of text, before editing

Just as long as there is an agreed format, and someone is in charge of coordinating the effort...

Subject: Re: Backup of the original Steam Forum Mon 17 Jul 2017 - 20:00

Ok - I´ll start next week with notepad++. When i need help, i will post it here.

Subject: Re: Backup of the original Steam Forum Mon 24 Jul 2017 - 18:51

The old forum contained about 26.250 posts, i think.
A search site had cached 105 pages with approximately 250 posts each.
I got 36 pages missing in between and 6 missing at the end.
So i only got 60% saved.
I will try to get the rest by searching the 1750 separately cached pages
containing approximately 15 posts each, if i can find.

If anyone wants to have the current result, send PM with your email adress.

Subject: Re: Backup of the original Steam Forum Mon 24 Jul 2017 - 19:49

Toony sent me what he had in .txt format. I combined all of the files into one large txt file, then began editing. As a proof of concept I threw a couple html tags on there for formatting and this is the result (its big, so wait for it to load):

[You must be registered and logged in to see this link.]

You can see where I began the editing, and where I stopped. The text document is much easier to read, even without my editing, but it may not be as easily searched as a google-indexed html page. It's going to take a long time, but I think adding the "------" with the breaks on either end to make it nice in html will really be the best, and simplest backup method for this.

Thoughts?

Oh, and here is the plain text file. You can see where I left off with the "-----" breaks.

[You must be registered and logged in to see this link.]

Subject: Re: Backup of the original Steam Forum Mon 24 Jul 2017 - 20:07

Wait, my friend - i found a way to get the missing pages.
In 2 hours you can have the complete old forum.

Subject: Re: Backup of the original Steam Forum Mon 24 Jul 2017 - 20:12

da_raabi wrote:: [You must be registered and logged in to see this link.]

Hi da_raabi, I am getting an Error 400 with this link.

da_raabi wrote:: Thoughts?
Oh, and here is the plain text file.

The plain text already looks impressive, so I look forward to seeing your improvements. It should be relatively straightforward to convert this to simple HTML pages. Google et al should find them easy to index.

Obviously a lot of work here, well done to you both!

Subject: Re: Backup of the original Steam Forum Mon 24 Jul 2017 - 20:32

Ok I see the error. It only shows up when I'm not logged into dropbox. I'll see if I can upload an html in a different way.

Toony - Awesome! I'll hold off on any additional edits until you get the complete text put together.

Once done, I'll do what I can to make a fairly readable and searchable file that maybe TB can upload on his site. I'm not 100% sure what this forum can handle, but if we can get the html uploaded somewhere, we should be able to do a site: Boolean search on the document to get at any info we may need.

Subject: Re: Backup of the original Steam Forum Mon 24 Jul 2017 - 21:16

da_raabi wrote:: Ok I see the error. It only shows up when I'm not logged into dropbox. I'll see if I can upload an html in a different way.

Toony - Awesome! I'll hold off on any additional edits until you get the complete text put together.

Once done, I'll do what I can to make a fairly readable and searchable file that maybe TB can upload on his site. I'm not 100% sure what this forum can handle, but if we can get the html uploaded somewhere, we should be able to do a site: Boolean search on the document to get at any info we may need.

You're both doing a brilliant job! Thank you very much indeed for this superb contribution to StalkerSoup Very Happy

Subject: Re: Backup of the original Steam Forum Mon 24 Jul 2017 - 21:26

da_raabi wrote:: if we can get the html uploaded somewhere, we should be able to do a site

I have a personal website - nothing to do with Stalkersoup, but I would be happy to put your HTML pages up there temporarily so you can see what they look like.

» Stalker Soup English Dialogue Project (from Steam)
» Forum rules
» Forum Wisdom Saves