How to automatically embed hyperlinks into citations and bibliographical entries of a downloaded PDF document! Part I

Nowadays, it is possible for a researcher to amass a huge number of electronic papers and books about his specialized area of interest. Naturally there are a lot of cross-references between these sources. The usual way of managing these files is the creation of a bibliographical database in one of the numerous relative programs like EndNote, Papers, JabRef and BibDesk. In this post I will try to present an exciting new way of "managing" your files which in a way makes managing them obsolete!

Specifically I created a workflow for embedding DevonThink hyperlinks into the very citations and bibliography entries of a PDF file. As a result when you want to read a cited paper or book you do not need to go to your bibliography manager program or even worse to the folder hierarchy of your hard drive in order to find the file. Instead you can just click the citation or the title or the author of the bibliography entry and the respective PDF file opens instantly! The magic of DevonThink hyperlinks in action!

The required tools/programs for this dream to come true are the following:

1. Adobe Acrobat Pro for Windows
2. AutoBookmark plug-in for Adobe Acrobat Pro
3. A Windows virtual machine 
4. DevonThink
5. BibDesk
6. TextMate
7. Link dictionaries
8. The AppleScripts for creating automatically the link dictionaries from BibDesk

More details about the workflow

AutoBookmark searches page text for occurrences of specific words or phrases and generates links annotations according to the user specifications. Link dictionary is a collection of search terms (words, phrases and etc.) and corresponding link actions. When it finds a search term from a dictionary it automatically creates a link annotation using a link action that is associated with it. Each search term should be unique within a link dictionary (excerpt from AutoBookmark help file). So you need to have an appropriate link dictionary so as the hyperlinks to the sources to be created automatically by AutoBookmark.

For our case I consider optimal the use of two different dictionaries each with its own advantages and disadvantages. The first contains  various possible forms of author-year citations e.g. Clark 2008, Clark (2008, Clark Stephen (2008, Clark S. 2008, Clark S. (2008. The advantage of this dictionary is that the matches are almost 100% unique. However the probability of the match is somewhat small because of the high variability in the way that the citations are made. The second contains the source titles. Now the situation is the complete opposite because you can very easily have false positive matches if the title is too generic but there is no case of a missing match if the title is adequately complex. 

In my next post I will describe how you can create the link dictionaries from your Mac.

How to convert a Mac-Unix style path to a Windows style path from Skim

Sometimes, I want to open a PDF file in my Windows virtual machine. To make things faster I wrote an AppleScript which converts the PDF file path of the front Skim window to its proper Windows style and then the converted path is sent to the clipboard.
As a result, all I have to do after the conversion is to paste the new path to the "File" field of the "Open" window in Adobe Acrobat Pro in Windows.

In order for this AppleScript to work properly, you need to customize the variable "WinPath" with the name and the drive letter of your hard drive.

The AppleScript code is the following:

tell application "Skim"
    set FilePath to get the path of front document
    set my text item delimiters to "/"
    set FilePath to text items of FilePath
    set my text item delimiters to "\\"
    set FilePath to (FilePath as text)
    set WinPath to "Z:\\MBProSystem" & FilePath
    set the clipboard to WinPath
end tell

How to connect a PDF file inside DevonThink with its record in BibDesk

I use BibDesk to manage and create my bibliographies but all my read and annotated PDF files are inside DevonThink. So I wanted an easy and fast way to connect the PDF files with their respective records in BibDesk. Technically I wanted to populate the "local URL" and "local File" columns of BibDesk. Consequently I wrote an AppleScript which does this job :-).

Moreover this AppleScript assigns the tag/keyword "Linked with DevonThink" to the record in which is applied. In order for this script to work properly you just have to select one PDF file in DevonThink and its respective bibliography record in BibDesk.

The AppleScript code is the following:

 --2011-03-08
--http://organognosi.blogspot.com
--the script works only for one PDF file per time

tell application "DEVONthink Pro"
    set this_item to the content record of think window 1
    set RecordLink to the reference URL of this_item
    set DevonThinkLink to RecordLink & "?page=0"
    set PdfPath to get the path of this_item
    set PdfPath to (POSIX file PdfPath) as alias
  
    tell application "BibDesk"
        activate
        set these_items to the selection of document 1
        repeat with theBook in these_items
            add DevonThinkLink to linked URL of theBook
            add PdfPath to linked files of theBook
            set currentKeywords to get keywords of theBook as string
            if (currentKeywords is not "") then
                set delimiter to ", "
            else
                set delimiter to ""
            end if
            set keywords of theBook to (keywords of theBook & delimiter & "Linked with DevonThink")
        end repeat
    end tell
  
end tell

Automated creation of a LaTeX compatible citation only from Skim! (with hyperlink included)

In the post "DevonThink links in Skim notes and AppleScript" I presented how you can create to a PDF file a number of Skim notes with various versions of references to itself. An example of these notes is shown below:

Then you can insert manually the BibTeX cite command in the fist and the second note.

Now you are ready to capitalize on these four little notes!

The first way was presented in the post "Automated summary creation linked with the source pfd file" where the respective AppleScript use the fourth note:

 set pdfDevonThinkLink to (get text for note 4 of page 1 of document 1) as string

In the AppleScript of this post the second note will be used.

As you understand the numbering of these notes plays a fundamental role in the effective use of these AppleScripts. For this reason it is required that you create these notes when there are no annotations in the first page of the PDF file. Otherwise these four notes will take another index number and you will need to adjust accordingly the AppleScript (2011-04-05, an alternative solution is to use the AppleScript in my post "How to create correctly the Skim notes which have the DEVONthink links, when you have already annotated the first page of the PDF document").

One of the most important reasons for which I have chosen to write my essays using LaTeX is that it makes possible the creation of rich hypertexts. By rich I mean that the final PDF document can have all the sorts of different kinds of links as you can see in the lower part of my mind map "The many faces of links". In order to create these links you should use the Hyperref package.

If you are accustomed to the LaTeX markup code you will recognize that the second note includes a proper citation as it is required by the Hyperref package:

(\cite{Deconstructing-the-Laws-of-Logic-Clark-2008a}: \href{x-devonthink-item://CDEC17B7-1EEE-42E1-B8C9-86A24C172BF3?page=}{})

The only things that are missing from the above citation are two numbers: the number for the DevonThink link and the the number of the referred page.

Now lets say that you would like to refer to the 27th page of the paper which is the third page of the PDF file. From now on you do not have to manually paste the necessary code and the appropriate numbers in your txt file but an AppleScript can do it for you!

There is only one but... you want the number that is shown in your text to be 27 but the number which is attached to the DevonThink link to be 2 (3-1: if you do not know why, read my post It's all about hyperlinks!). My AppleScript has the ability to recognize in which page you currently are but it cannot recognize the written number in the page! So you will have to enter it manually.

 The AppleScript code is the following:

--2011-03-07
--http://organognosi.blogspot.com

display dialog "Give the number which is written in you current page:" default answer "" buttons {"Cancel", "OK"} default button 2
set writtenNumber to text returned of the result as text

tell application "Skim"
    activate
   
    tell document 1
        set currentPageNumber to get index of current page
        set pageDevonThinkNumber to currentPageNumber - 1
        set pageDevonThinkNumberText to pageDevonThinkNumber as text
        set DevonThinkLink to get text of note 2 of page 1
       
        tell application "TextMate"
            activate
            insert DevonThinkLink
            tell application "System Events"
                key code 123
                key code 123
                key code 123
                key code 123
            end tell
            delay 0.5
            insert pageDevonThinkNumberText
            tell application "System Events"
                key code 124
                key code 124
            end tell
            delay 0.5
            insert writtenNumber
        end tell
    end tell
end tell

If you do not want to write even this number you can create a fifth note which will have only the written number of the first page and then the AppleScript can compute the rest.

--2011-03-07
--http://organognosi.blogspot.com

tell application "Skim"
    activate
   
    tell document 1
        set currentPageNumber to get index of current page
        set pageDevonThinkNumber to currentPageNumber - 1
        set pageDevonThinkNumberText to pageDevonThinkNumber as text
        set DevonThinkLink to get text of note 2 of page 1
        set writtenNumberOfFirstPage to (get text for note 5 of page 1) as string
        set currentPageWrittenNumber to currentPageNumber + writtenNumberOfFirstPage - 1 as string
       
       
        tell application "TextMate"
            activate
            insert DevonThinkLink
            tell application "System Events"
                key code 123
                key code 123
                key code 123
                key code 123
            end tell
            delay 0.5
            insert pageDevonThinkNumberText
            tell application "System Events"
                key code 124
                key code 124
            end tell
            delay 0.5
            insert currentPageWrittenNumber
        end tell
    end tell
end tell

Finally the easiest case is when the page numbers of the PDF file and the printed book coincide. In this case you can still use the above AppleScript if you write in the fifth note the number 1. If you do not want to use a fifth note the AppleScript is the following:

--2011-03-07
--http://organognosi.blogspot.com

tell application "Skim"
    activate
   
    tell document 1
        set currentPageNumber to get index of current page
        set pageDevonThinkNumber to currentPageNumber - 1
        set pageDevonThinkNumberText to pageDevonThinkNumber as string
        set currentPageNumberText to currentPageNumber as string
        set DevonThinkLink to get text of note 2 of page 1
       
       
        tell application "TextMate"
            activate
            insert DevonThinkLink
            tell application "System Events"
                key code 123
                key code 123
                key code 123
                key code 123
            end tell
            delay 0.5
            insert pageDevonThinkNumberText
            tell application "System Events"
                key code 124
                key code 124
            end tell
            delay 0.5
            insert currentPageNumberText
        end tell
    end tell
end tell

Before you execute any of the above AppleScripts you should have a TextMate document opened. Moreover the cursor should be at the point in which you want the citation to be inserted.

2011-04-05
There is an improved version of the above AppleScript in my post "Automated creation of references with hyperlinks from Skim"

Open a DevonThink hyperlink directly to Skim

I use a QuicKeys macro in order to open a DevonThink PDF hyperlink directly to the specific PDF page in Skim. In the following image you see the required steps of the macro. In the third step you should put the whole AppleScipt from my previous post:  "Go from the current PDF page in DevonThink to the same page in Skim - AppleScript" 

Then you can assign a specific gesture (for example "four finger click") using BetterTouchToul to the used hot key (option + cmd + F1) for the macro and you are done! Now you can bypass DevonThink and read the linked PDF page in Skim immediately!

You can download this macro from here (right click + Sava as)

 Automation rules!

Go from the current PDF page in DevonThink to the same page in Skim

I prefer reading my PDF files in Skim. Moreover all my read PDF files are inside DevonThink databases and all my PDF hyperlinks lead to DevonThink. As a result I regularly used the "Open in external viewer" function of DevonThink. Unfortunately the PDF is opened in the first page so you have to manually go to the desired page. If you need to do this a lot of times after a while it becomes annoying. So I wrote an AppleScript which automates this procedure :-).

The AppleScript is the following:

tell application "DEVONthink Pro"
    set pdfFile to the content record of the think window 1
    set RecordLink to the reference URL of pdfFile
    set PdfPage to current page of the think window 1
    set DevonThinkLink to RecordLink & "?page=" & PdfPage
    set the clipboard to DevonThinkLink
    set PdfPath to get the path of pdfFile
    set PdfPath to (POSIX file PdfPath) as Unicode text

   
   
    tell application "Skim"
        open file PdfPath
        set PdfPage to PdfPage + 1
        go document 1 to page PdfPage of document 1
    end tell
end tell

Automated summary creation linked with the source PDF file

One of the great characteristics of Skim highlights is that they capture automatically the highlighted text  in a newly created note.

Moreover you can export these notes in a separate txt or rtf file.

As a result an approximation of a summary is created if you highlight the appropriate text passages.

Additionally if you have created the Skim notes with the DevonThink links as I described in the post "DevonThink links in Skim notes and AppleScript" you can use another AppleScript in order to have a properly formatted hyperlink from every highlighted text to the exact pdf page of the paper or book.

When you copy this text to the respective wiki page of the paper or book you will have a nice hypertext between your summary and the source pdf file.

You can see the seamless integration between the summary and the pdf in the following YouTube video.

The AppleScript code is the following (updated):

 tell application "Skim"
    set pdfDevonThinkLink to (get text for note 4 of page 1 of document 1) as string
    set pdfDevonThinkLinkWihtoutZero to (text 1 thru ((length of pdfDevonThinkLink) - 1) of pdfDevonThinkLink) as string
  
    set allMyPages to pages of document 1
    set numberOfMyPages to length of allMyPages as integer
  
  
    tell application "TextMate"
        activate
        tell document 1
            insert "===Skim notes===
"
        end tell
    end tell
  
    repeat with currentPage from 0 to numberOfMyPages
        set pageNotes to notes of page currentPage of document 1
        set numberOfPageNotes to length of pageNotes as integer
      
        repeat with currentNoteNumber from 1 to numberOfPageNotes
            set pdfText to (get text for note currentNoteNumber of page currentPage of document 1) as string
          
            set pageForDevonThinkLink to currentPage - 1
            set the clipboard to pdfText
          
            tell application "TextMate"
                tell document 1
                    insert (the clipboard) & " [" & pdfDevonThinkLinkWihtoutZero & pageForDevonThinkLink & " p. " & currentPage & "]

"
                end tell
            end tell
        end repeat
    end repeat
end tell

In order for the script to work you need to have TextMate installed in your Mac and an empty txt file open.

How to put DevonThink links in Skim notes

For every book or paper that I read I create a new page in my wiki. The name of this page is created under the following template: Title - Author, for example "Deconstructing the Laws of Logic - Clark". In this page I keep all my notes and my thoughts concerning this paper. The pdf file is inside a DevonThink database. When I am reading a pdf I want to be able to use the DevonThink link without leaving Skim. To this end I wrote an AppleScript, which creates in the first page of a pdf four Skim notes, containing four different ways of constructing references using DevonThink links.

 

The fourth note is the simplest and contains the DevonThink url (x-devonthink-item://CDEC17B7-1EEE-42E1-B8C9-86A24C172BF3?page=0). Its extended text contains the title of the wiki page. 

The third note contains a MediaWIki internal link to the wiki page of this pdf followed by an external link to the pdf file itself.

The second note contains the necessary Latex code for creating a proper bibliography reference and a link to the exact pdf page using the hyperref package. The only things that are missing from the above code are the cite key (between the opening parenthesis and the colon) and the page numbers. As an example the complete second note is the following: 

(\cite{Deconstructing-the-Laws-of-Logic-Clark-2008a}: \href{x-devonthink-item://CDEC17B7-1EEE-42E1-B8C9-86A24C172BF3?page=}{})

As you can notice the cite key "Deconstructing-the-Laws-of-Logic-Clark-2008a" is quite long. I have chosen this because I want to recognize the reference just by looking in its cite key and not by trying to remember for example which paper Clark wrote on 2008...

The code for a bibliography reference to the third page of this paper which is in the 27th page of the Philosophy Journal is the following:

(\cite{Deconstructing-the-Laws-of-Logic-Clark-2008a}: \href{x-devonthink-item://CDEC17B7-1EEE-42E1-B8C9-86A24C172BF3?page=2}{27})

The first note contains the necessary MediaWiki code for creating a link to the excact pdf page which is compatible with LaTeX whenever you want to export the wiki text to Latex (more about this in a quite distant in time post...)

Now every time I want to create a link to a pdf file everything is at hand in one of these notes! 

The AppleScript code which makes all the above possible is the following: 

tell application id "com.devon-technologies.thinkpro2"

    set these_items to the selection
    if these_items is {} then error "Please select some contents."
   
    repeat with this_item in these_items
       
        set RecordLink to the reference URL of this_item
        set DevonThinkLink to RecordLink & "?page=0"
        set PdfPath to get the path of this_item
        set PdfName to the name of this_item
       
       
        tell application "Skim"
            open PdfPath
            tell page 1 of document 1
                make note with properties {type:anchored note, bounds:{523, 820, 540, 820}, text:"(<rawtex></rawtex>: [" & RecordLink & "?page=])", extended text:DevonThinkLink}
                delay 0.1
                make note with properties {type:anchored note, bounds:{523, 800, 540, 800}, text:"(: \\href{" & RecordLink & "?page=}{})"}
                delay 0.1
                make note with properties {type:anchored note, bounds:{523, 780, 540, 780}, text:"([[" & PdfName & "]]: [" & RecordLink & "?page=])"}
                delay 0.1
                make note with properties {type:anchored note, bounds:{523, 760, 540, 760}, text:DevonThinkLink, extended text:PdfName}
            end tell
            save document 1
            delay 1
            close document 1
           
        end tell
    end repeat
end tell

2011-04-02 
In my post "How to create correctly the Skim notes which have the DEVONthink links, when you have already annotated the first page of the PDF document" there is an improved version of the above AppleScript. The newer one solves a problem which appears when you want a third AppleScript to use the aforementioned Skim notes like the AppleScript in my post "Automated creation of a LaTeX compatible citation only from Skim! (with hyperlink included)". Moreover it creates a fifth note for which there is detailed information in my post "Latin page numbers, Arabic page numbers and the fifth Skim note".

A road map for my blog - The many faces of links

In the following mind map you can see the various kinds of links that I am able to use and the respective programs that are required in order to create each of them. Moreover I have added some hyperlinks in the nodes for which I have posted relevant information.

From A Digital Workflow for Academic Research

You can download this mind map in its native Mind Manager file format from here  and as a PDF from here (right click + Save as).

Finally, you can view this mind map as a roadmap for my blogspots... post by post I will try to present how you can create and use these different ways of cross-linking your thoughts, notes, writings and also your sources into an integrated whole!  

How to use DevonThink links in a MediaWiki page

MediaWiki supports out of the box a number of protocols for external linking such as HTTP, HTTPS, mailto, IRC. Additionally it gives you the possibility to add your own protocols by modifing the LocalSettings.php file under the root folder.

I have installed MAMP in order to use MediaWiki and lets say that my wiki is named "MyWiki". Then the path for this file in my file structure is (I use Snow Leopard):

/Applications/MAMP/htdocs/MyWiki/LocalSettings.php

So in order for MediaWiki to be able to understand DevonThink links you need to add the following line to the LocalSettings.php file:

$wgUrlProtocols[] = 'x-devonthink-item://';    

Then when you write in the edit mode of a MediaWiki page:

you just see

a nice hyperlink after saving. 

Now MediaWiki and DevonThink are perfectly integrated! See a short demonstration in the following video.

If you want to learn how the wiki text in the above video can be created automatically using Skim, AppleScript and Textmate read the following posts:

• DevonThink links in Skim notes and AppleScript
• Automated summary creation linked with the source pfd file

It's all about hyperlinks!

If I was forced to choose the single most important element of my workflow then my answer would be the creation and usage of hyperlinks. We are all accustomed of using hyperlinks in the World Wide Web but I have observed that the usage of them in the personal mode of working is quite rare.

A classic but somewhat outdated form of link is the bibliography reference that is commonly used in academic texts. Its main disadvantage is that you need a lot of time and energy to actually find and use the referred passage. Nowadays a different situation is possible by using hyperlinks.  

I chose to use MediaWiki, DevonThink and Latex because they make the cross-linkage of my notes, my sources and my writings very easy and fast.

Specifically lets say you have A and B wiki pages. If you want to create a hyperlink from A to B you just have to write [[B]] in the wiki text. The whole process is very well documented in the MediaWiki help page about links.

The respective linking process with DevonThink is not well known and I think it is quite important. I even believe this capability of DevonThink to be a reason for academic researchers to switch to Mac computers!

The DevonThink program assigns a custom Uniform Resource Locator (URL) for every file that is imported or indexed to it. An example of this URL is the following:

"x-devonthink-item://6632BE51-AEDF-42F9-9CFD-BFED462EBC33"

Moreover, in the case of a pdf file, a different URL is assigned to every single page of it. This URL takes the form:

"x-devonthink-item://BB2D2CDA-0BCA-4F72-8C8E-64B49CA12839?page=0"

As a result when you want to refer to the twenty fifth page of a pdf document you just have to use the URL:

"x-devonthink-item://BB2D2CDA-0BCA-4F72-8C8E-64B49CA12839?page=24"

Notice that the page number in the URL is 24 and not 25 as the index base that is used by DevonThink is zero and not one.

You can send this URL to the clipboard by clicking "Copy Page Link" in the contextual menu.

Then you can use this link in order to create a hyperlink to the exact pdf page! As a result all your bibliography references can be transformed to live links to the respective pages. No more you have to hunt down the source of a quotation!

There is only one but...

Not every word processor accepts the DevonThink links, the most prominent of them... Word.  This is an additional reason that I do not use Word...


Some programs that are fully compatible with the DevonThink links are the following:

• DevonThink rich text editor
• TextEdit
• Nisus Writer Pro
• Skim notes

In the above programs after you paste a DevonThink link you do not see the raw URL but the name of the linked file!

Then you can edit the link as you like...

Be careful only not to start the editing from the first or the last character of the file name.
You can even edit the DevonThink link if you want to refer to a different page of the same pdf file.

But the best thing is that you can use DevonThink link inside MediaWiki pages and Latex. If you want to learn how read my next post...

Some tools of the trade

I use the following hardware:

• a MacBook Pro 17in
• a MacMini (as a server for my wikis and my ebooks library)
• an iPad
• a Time Capsule
• an iPhone 3G
• several external hard drives (for backup)
• a 27in LG monitor

In my academic workflow I use the following programs:

• Skim: for reading papers and books (in pdf format)
• Calibre: for organizing all my ebooks (read and unread)
• DevonThink: for organizing everything that I have read
• BibDesk: for reference management and automated bibliography creation
• MediaWIki: for notes management
• TextMate and LaTex: for writing my papers
• AppleScript: for automating my workflow