Board index » The Property Pin Website - Open Development » theideapin.com - Open Dev

Post new topic Reply to topic  [ 20 posts ]  [Go to page]   1, 2  Next
Author Message
 Post subject: Scraping property pin posts
PostPosted: Fri Dec 05, 2014 1:32 am 
Offline
Migrant

Joined: Nov 30, 2014
Posts: 2
Hi everyone,

I'm a computer science student at TCD and I'm interested in doing a project that analyzes posts from thepropertypin forum.
For my project, I need to scrape some posts from this website and I am just wondering if it would be okay if I did this?

Are there any admins I can contact about this? I sent an email to openwindow but I don't think he/she checks that account, or it went to spam. (Also, I presume there's a minimum number of posts needed before pms can be sent as I can't send pms)

I don't really want to give away too much about what I want to research, because it may skew the results(it probably won't, but better safe than sorry).

Also, I can publish the results online after my project is done, which will be some time around April/May :)

Thanks


Last edited by What Goes Up... on Fri Dec 05, 2014 1:36 am, edited 1 time in total.
Removed poster's firstname from bottom of OP


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Fri Dec 05, 2014 1:40 am 
Offline
Nationalised
User avatar

Joined: Jun 24, 2008
Posts: 10232
Location: South of the Border (Down Mexico Way)
My 2¢:

It's in the public domain.


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Fri Dec 05, 2014 4:13 am 
Offline
Planning Tribunal Attendee

Joined: Feb 19, 2014
Posts: 1380
Have at it, but don't quote me on that, my first ambition in life is to be an anonymous source, my 2nd, to be a member of a cabal.

I believe the National Library operates a scraping racket. Not sure about the extent or limitations.


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Fri Dec 05, 2014 4:31 am 
Offline
Of Systemic Importance

Joined: Jun 9, 2008
Posts: 7093
Maybe get a life instead.

_________________
“Don't ask the barber if you need a haircut—and don’t ask an academic if what he does is relevant.” Nassim Nicholas Taleb


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Fri Dec 05, 2014 9:54 am 
Offline
Planning Tribunal Attendee

Joined: Feb 4, 2011
Posts: 1482
Location: A rock drifting aimlessly in the north atlantic
grumpy wrote:
Maybe get a life instead.


Seriously? A guy comes looking for assistance and info and that is your response.

@outterest
I can see no reason as to why you should not scrap them, people have been scraping forums for quite a while.
My only concern would be around personally indicative information a poster may have left on the forum. I believe that If you are doing this as part of a thesis, it is up to you to ensure that personally indicative information is concealed/redacted, but I'm not sure if this applies to info in the public domain, so I would advise a word with your tutor in this regard.

_________________
"Don't let it end like this. Tell them I said something...."
Last words of Pancho Villa


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Fri Dec 05, 2014 10:17 am 
Offline
Planning Tribunal Attendee

Joined: Oct 7, 2014
Posts: 1010
grumpy wrote:
Maybe get a life instead.


Some of us managed to get a life and a degree...

EDIT: I think it's good form (as you have done) to ask, but I don't think you really need permission, as such.


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Fri Dec 05, 2014 10:25 am 
Offline
Nationalised
User avatar

Joined: Feb 9, 2010
Posts: 9019
Perhaps Grumpy and the admins can see something interesting in the IP address? (Although I'm not sure who would be at work at 12.32 AM)


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Fri Dec 05, 2014 12:40 pm 
Offline
Of Systemic Importance

Joined: Jun 9, 2008
Posts: 7093
I was somewhat tired and emotional and sincerely apologise.

_________________
“Don't ask the barber if you need a haircut—and don’t ask an academic if what he does is relevant.” Nassim Nicholas Taleb


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Fri Dec 05, 2014 12:49 pm 
Offline
Property Magnate

Joined: Nov 8, 2006
Posts: 514
grumpy wrote:
I was somewhat tired and emotional and sincerely apologise.


You were just plain old grumpy :twisted:


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Fri Dec 05, 2014 3:41 pm 
Offline
Too Big to Fail

Joined: Jun 25, 2006
Posts: 3233
bedsit wrote:
grumpy wrote:
I was somewhat tired and emotional and sincerely apologise.


You were just plain old grumpy :twisted:


Awwh, give him a break, after all, statistics tell us that just over 85% of all dwarves are not Happy and that one in seven is in fact Grumpy.

:wink: :-GC



Blue Horseshoe

_________________
"Housing traditionally is not viewed as a great investment. It takes maintenance, it depreciates, it goes out of style". "So, why was it considered an investment? That was a fad. That was an idea that took hold in the early 2000's. And I don't expect it to come back. Not with the same force. So people might just decide, "I'll live in a rental." That is a very sensible thing for many people to do". Prof. Robert J.Shiller, Nobel Laureate in Economics, Feb 2013.


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Fri Dec 05, 2014 4:31 pm 
Offline
Too Big to Fail

Joined: Aug 22, 2008
Posts: 3549
https://github.com/lairdshaw/fups

May be useful? I recall in the past I used it and came up against an issue in recognising custom syntax on this forum due to customisations done. But more work has been done on the code since I understand. Maybe it was fixed, maybe not. And all contributions to the code welcome etc.


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Fri Dec 05, 2014 11:45 pm 
Offline
Nationalised
User avatar

Joined: Jan 1, 1970
Posts: 22908
I knew this day would come! Can I retire? Do you have an alcoholrithm to correct my historical typos? If so you might have some leverage. XD

I'm sure the technical answer is no. The real answer why not. The political answer is we had a solemn understanding that the data was secure. The Garda press release, we believe contraband water was on the premises, deadly force was required to liberate the asset.

.... I'll check the email.

_________________
Follow The Pin - https://twitter.com/dailypinster

"Politicians are always realistically maneuvering for the next election. They are obsolete as fundamental problem-solvers." - Buckminster Fuller

"I was comfortable with a couple of banks being married today, instead i wake up and find I'm married to the banks." - Catbear

Image


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Sat Dec 06, 2014 3:06 am 
Offline
Migrant

Joined: Nov 30, 2014
Posts: 2
roc wrote:
https://github.com/lairdshaw/fups

May be useful? I recall in the past I used it and came up against an issue in recognising custom syntax on this forum due to customisations done. But more work has been done on the code since I understand. Maybe it was fixed, maybe not. And all contributions to the code welcome etc.


Thank you! I was looking at using BeautifulSoup with python in the last few days, but this might save me doing extra work.


The main reason I'm asking permission is because I'll be analyzing user posts and possibly categorizing users based on what they've said. That might sound a bit sinister - especially since I'm not saying what I'm using it for, but I'm not using it for anything even remotely similar to advertising or selling the information to companies.

Anyway, as some have said, it's in public domain, so that sort of thing could be done anyway if someone felt so inclined...

Thanks for the support :)


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Sat Dec 06, 2014 9:04 am 
Offline
Nationalised
User avatar

Joined: Jan 4, 2013
Posts: 17326
Location: To the right of the decimal place
Presumably you have a research proposal. Why don't you post it? It's not like it's going to retrospectively change what people post.

Also note that the Piston area exists in a limbo area - it's purposfully not scrapabke by google etc. Of course you could register an account to scrape it but that would seem like bad form to me.

_________________
— Try, fail, understand, win. —


Top
 Profile  
 
 Post subject: Re: Scraping property pin posts
PostPosted: Sat Dec 06, 2014 9:28 am 
Offline
Under CAB Investigation

Joined: Mar 31, 2009
Posts: 1799
This type of exercise could have been done 100 times by less polite researchers.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 20 posts ]  [Go to page]   1, 2  Next

    Board index » The Property Pin Website - Open Development » theideapin.com - Open Dev

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Jump to:  

Follow, Retweet @dailypinster



Pyramid Built, Is Better Built! - Latest Property Discussions www.thepropertypin.com