XPath Data Scraping From Online Community Using Excel

I recently read this article on how to scrape the Inbound.org community members profile using Excel. And you can watch the video here if you prefer it that way.

Since the release of this tutorial, the Inbound website structure has changed a bit, as you can see at minute 11:00 in the video, if you attempt to copy the XPath of the social media icons it appears slightly different and because of this I haven't been able to extract that information.

Here's what I get now:

/html/body/div[3]/div/div/div[1]/div/div[2]/a[1]/i

This is how I wrote the syntax in Excel:

=XPathOnUrl(A2,"//a[@class='twitter']","href")

And then like this:

=XPathOnUrl(A2,"//a[contains(@class,twitter)]/@href")

Although I tried in many different ways, none of them showed me the link to the member's social media profile.

I even tried changing the xpath in multiple ways to get different data from the page, but none of it was the social media information:

=XPathOnUrl(A2,"//*[contains(@class,member-banner-tagline)]/div[2]/div/div/div[1]/div/div[1]")
=XPathOnUrl(A2,"//*[contains(@class,member-banner-tagline)]/div[2]/div/div/div[1]/div/h1")

I honestly don't know what to try anymore, something's wrong and I can't figure it out. Anybody have enough experience with this or can pinpoint the problem here with my syntax?

Thanks a lot

1 Answer

According to Firefox/Firebug, the unique XPath to a users twitter URL is:

/html/body/div[2]/div/div/div[1]/div/div[2]/a[1]

Which is slightly different to your first attempt. NB: I've amended my first attempt, you don't want the /i on the end or you get the inner element not the <a> with the url on it.

But then, your second attempt certainly finds the correct HTML element that contains the URL and it is more robust.

Your 3rd attempt doesn't work with XPathOnUrl though it works in Google Docs ImportXML because it doesn't support the attribute syntax.