Category Archives: Code

Apple hijacks Unix headers into Xcode in Mavericks

I am currently taking a class on Unix systems programming. While following along with lecture, the professor stated that almost always the header files needed for our type of work are located at /usr/include. However, that directory does not exist on my brand new Mavericks MacBook Pro. I have learned (after much searching the web) that the header files are now here:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr

After doing some more research, I found that to get those Unix header files back into /usr/include, one has to install Xcode’s command line tools. But, in order to get those, one has to be a registered member on Apple’s developer website. Not necessarily a paying member, but registered.

I was once a paying member for Apple’s developer tools but I gave it up because I was not actually using everything that was made available. My focus changed and the annual $99 was going to waste. In fact, because of that and that Apple has so much of their Cocoa documentation available externally for free, my need to log into their website has decreased over time to my not touching it in over a year. Now I am jumping through the hoops to figure out which account I was using and what was the password, but that is turning out to be harder than expected for a variety of reasons on Apple’s side, the servers not propagating my Apple ID resets to the developer site being one of them. Yes, I should have done a better job of recording my information, but a simple password reset shouldn’t be this hard either. At this point, I will likely just create a new account solely for getting me what I need.

I have no idea when Apple hijacked the header files, and i can understand the logic and convenience of doing so for tool updates, but knowing what little I do about Unix, hijacking seems to be anathema to the Unix culture. Apple has made my morning nothing but hassle trying to get this fixed and I will have an assignment due soon. So, cheers for that, Apple.

UPDATE: I wound up creating an account solely for the developer account, and it was, as expected, faster and easier than mucking about with a bunch of password resets. Once I did that, getting the command line tools was not straightforward though not hard (Xcode > Open Developer Tool > More Developer Tools… which then kicks you over to a downloads page on the developer website, with registration required for access).

As for my comment about the new default header location being anathema to Unix, I realize now that’s not necessarily true. If anything, Apple can do whatever they want with their distro. But the fact remains based on everything I have read so far that there are some clear expectations about there things ought to be and the header file location is one of them. But at least now I can establish a workflow where I can use the muscle of Xcode to develop and then confidently test outside before submission.

Applescript: Getting unique items in a list update

I love getting questions about the contents of, or topics related to, my site. Most recently, I was emailed a question about one of the older functions I have in the Applescript section. In particular, it was one for getting unique items in a list. Here’s the function…

on GetUniqueItems(sourceList)
	set itemCount to (get count of items in sourceList)
	set compiledList to {}
	--get the first item to kick off the list
	repeat with x from 1 to itemCount
		set itemFound to false
		set itemX to item x of sourceList
		if x < itemCount then
			repeat with y from (x + 1) to itemCount
				set itemY to item y of sourceList
				if itemY is itemX then set itemFound to true
			end repeat
		else
			repeat with y from 0 to (itemCount - 1)
				set itemY to item y of sourceList
				if itemY is itemX then set itemFound to true
			end repeat
		end if
		if itemFound is false then
			set end of compiledList to itemX
			exit repeat
		end if
	end repeat
	--if no items are found
	if (get count of items in compiledList) is 0 then
		return compiledList
	end if
	--find the rest of the unique items
	repeat with x from 1 to itemCount
		set itemFound to false
		set itemX to item x of sourceList
		set resultCount to (get count of items in compiledList)
		repeat with y from 1 to resultCount
			set itemY to item y of compiledList
			if itemY is itemX then set itemFound to true
		end repeat
		if itemFound is false then set end of compiledList to itemX
	end repeat
	return compiledList
end GetUniqueItems

The question was focused on why I go through the source list more than once. As soon as I saw the function after the question, I knew they were right that something was wrong. My answer essentially explained that this was one of the first useful home-brewed functions I wrote, and since it worked, it stuck, as working code is wont to do. But, honestly, I’ve reviewed this code a dozen times and it has me completely baffled as to how it works. I think there is even a whole block on there that can come out and nothing would change.

I started writing my first Applescripts in 2005, which was also my first serious foray into programming. The last time I thought about if...then statements was in high school writing BASIC for the Commodore 64 in high school. This function, according to my notes, was written in 2007 when my needs and skills were becoming more robust. This function is currently in use in several scripts today with nary an error. But, nine years of experience later and immediately that function is absolutely cringe-worthy (though only to a point considering when I wrote it), so I rewrote it. Et voilà…

on getUniqueItems(src)

	set srcCount to (count src)

	set unq to {}

	repeat with x from 1 to srcCount

		set srcItem to item x of src

		set unqCount to (count unq)
		set match to false

		repeat with y from 1 to unqCount
			set unqItem to item y of unq
			if srcItem = unqItem then
				set match to true
			end if
		end repeat

		if match is false then
			set end of unq to srcItem
		end if

	end repeat

	return unq

end getUniqueItems

Hindsight being 20/20 and all that, this a “duh!” moment. There are a couple important things to note about this.

First, my test data for these types of functions is reliable but small. This is O(n2) on the low-end of things, but almost invariably Applescripts very rarely ever deal with data sets large enough where O(nk) has enough of an impact to get a coffee and sandwich while waiting. My personal experience and preference is that if that were the case, then I need to go find a more appropriate tool for data prep.

Second (and last), this block…

set unqCount to (count unq)
set match to false

repeat with y from 1 to unqCount
	set unqItem to item y of unq
	if srcItem = unqItem then
		set match to true
	end if
end repeat

if match is false then
	set end of unq to srcItem
end if

…could be replaced with this common Applescript hook…

if unq does not contain srcItem then
	set end of unq to srcItem
end if

The only problem with this, as I see it, is when trying to compare custom data types as opposed to core data types. This is great if I only ever worked with Applescript’s core data types, like string, number, date, and the like. But almost all of my Applescript code has been targeted to Adobe’s Creative Cloud, which brings a wealth of custom objects with loads of properties with which to work. I think leaving in the extra code (and any possible hits on speed since this not baked into the language like contains) is reasonable for the sake of easy customization later. By way of example, this…

-- compare memory addresses
if srcItem = unqItem then

…becomes this in a pinch…

-- compare object properties
if foo of srcItem = foo of unqItem then

…or even…

-- deep comparison
if my customCompare(foo of srcItem, foo of unqItem) then

So, a bit of extra code for the win. I suppose I could set up a hash table implementation to improve upon the O(n2)O(nk) range of complexity, but with Applescript work, again, it’s really not worth it.

That was a really great question on a number of levels. Not just that this shows that people actually read the site on occasion and finds something useful, which is the core goal of the site (this blog is really more of just a place to vent that offers me more flexibility than other blogging sites or social media) but also to be compelled to review and improve old code and find just how far I have advanced over the years. Win-win.

“And in truth, I’ve never known a man worth his salt who, in the long run, deep down in his heart, didn’t appreciate the grind, the discipline. The difference between a successful person and others is not a lack of strength, not a lack of knowledge, but rather… a lack of will.”
Vince Lombardi

Studies in Semicolons: The Parable of the Carpenter

The carpenter understands the value of something he works with every day, and that’s why he spends so much money on the hammer. But he also understands that value is a double-edged sword: he’s committing to the product he knows, that is reliable.
Studies in Semicolons: The Parable of the Carpenter

Replacing the subject of the punchline with other tools in which I have invested makes this parable applicable to more areas than I care to think about. Interestingly enough, Microsoft Office is not one of them.

Xcode, Mavericks, and OSA

It would appear that Xcode in Mavericks now shows certain Scripting Bridge warnings in the console by default. I upgraded to Mavericks Friday, started a debug run of a SB-using project and this appeared out of nowhere:

2013-10-28 09:14:42.849 Ratchet[2387:303] warning: failed to get scripting definition from /Applications/Adobe Photoshop CC/Adobe Photoshop CC.app; it may not be scriptable.
2013-10-28 09:15:08.048 Ratchet[2387:303] warning: failed to get scripting definition from /Applications/Adobe InDesign CC/Adobe InDesign CC.app; it may not be scriptable.

This appears to be neither actionable or able to be disabled, so not much use during runtime. Not that it matters to me anymore anyway.

Not quite as simple as all that

Nigel Warren:

The fact that iWork on the Mac has lost functionality isn’t because Apple is blind to power users. It’s because they’re willing to make a short-term sacrifice in functionality so that they can create a foundation that is equal across the Mac, iOS, and web versions. It will take time to bring these new versions of iWork up to parity with what the Mac used to have. In the meantime all platforms have to live with the lowest common denominator.

This is what I think, too. Doesn’t make it any easier to stomach if you relied on features that have gone away though. And let’s see how long “short-term” is.
Daring Fireball: iWork 13 is the New iMovie 08

This analogy falls short when discussing Applescript’s lack of support in iWork and its oftentimes murky support from Apple and its partners. From an application/GUI standpoint, sure, iWork is garnering the same reaction as iMovie and Final Cut Pro. Not that I want to point out the obvious here, but Applescript is not a GUI. Applescript is a programming language in its own right that extends the capability of any supporting application to well beyond the confines of its GUI. Looking at the current state of Applescript support is important to take into consideration here because there exists a deeper, more complex, history than what exists for iWork.

  • Search “Applescript” on Apple’s homepage, and there is no evangelism of any kind. There are, instead, links to developer documentation. What evangelism did exist has long since been migrated over to an external site run at one point by a freelancer specializing in Applescript solutions but now any clue of ownership is now hidden. What, exactly, is the Applescript community is to think of this? While there is nothing from Apple stating that Applescript support is going away, but there is an implication that Applescript is less than a second-class citizen in the Apple world, simply sent out to pasture and brought in for the occasional grooming. Attempts to improve Applescript’s profile are either weird or poorly implemented.
  • ApplescriptObjC was created to allow Applescript developers to create Cocoa applications in native Applescript, but it has some serious flaws. The source code file name changes from .applescript to .scpt upon compilation which destroys any links to included files. This is ultimately manageable but that just creates more ceremony before development can begin. Also, blocks are cannot be implemented in ApplescriptObjC, either.
  • Scripting Bridge, which goes the other way by allowing Applescript calls via native Objective-C (as opposed to using the NSApplescript class). But it relies on two command line apps that generate grossly erroneous or incomplete header files (with no tools for troubleshooting issues) and typically results in an API that is severely limited in scope by randomly not allowing the creation of certain classes.
  • Automator appears to be a bridge between budding power users that can’t code to those that can, but is a separate implementation in addition to Applescript despite both use the same frameworks. More work to implement another niche technology? Really?
  • Now we have a major name on the Mac, Adobe, allowing their Applescript APIs to fall apart and currently not allowing external script calls.

Given that context, iWork’s change in Applescript is not “the sky is falling” but rather the latest in a long line of dysfunctional behaviors from Apple.

If an analogy is sought of Applescript’s drop from iWork, then it could go something like this: For a certain class of “power users” (an analogy I equate to “people who drive trucks”), dropping Applescript support from a core application cutting off any further independent development and is akin to Apple dropping the Cocoa frameworks and Objective-C. Hard stop. There is no GUI to fall back on for what Applescript does for a lot of its users.

For that subclass of power users, those that have built processes and workflows that save their companies thousands of dollars and hours each year in productivity, any drop in Applescript support marginalizes those users into maintaining quickly-outdated systems. When talking about those power users, adding a “though” to “Doesn’t make it any easier to stomach if you relied on features that have gone away” is dismissive of the impact. The reaction from certain users is not about iWork but about the state of Applescript in general.

Given this affects a subclass of a subclass of all users, robust Applescript implementations are a relatively small niche. As such, support was never as solid or apparent as, say, Core Data. If support for a technology that I use to automate quantifiably money-saving processes is dodgy to the point where whole code migrations are required, then I, as well as others, will need to seek out more stable platforms. So, try to stomach that.

Wither Applescript

John Gruber linked to an article that explains the dropping of Applescript from the latest release of iWork.

The writing has been on the wall about this for a long, long time. How long has it been since Apple allowed their Applescript pages be allowed to go offsite? Instead is simply lamenting about the impending, far-too-long-drawn-out demise of Applescript, I wanted to substantiate a claim made by the aforementioned article:

What I suspect Apple doesn’t realize is how much small business and small shops workflow depends upon Applescript. Casual use is fine. But a lot of people do more.
iWork 13 — A Huge Regression

iWork is irrelavent in my industry; the lingua franca for developing book content is Microsoft Office, but the red flag being raised is just as relevant. Also, I am not a small company. I work for a large company that is almost a third Macs. But, I think there is the point to be made that Apple might not be aware of the complexity of workflows built around Applescript, and how casually and quietly deprecating support sucker punches any developer regardless of size.

Background

At my day job in a publishing company, I manage a pre- and post-press department that is concerned with the archiving, retrieval, and most importantly re-use of content. (I suppose this could be summarized as “content lifecycle management.”) Content comes in many forms but for me the focus is on art and text.

For art, we have a number of digital products that are derived from the main texts that we publish, all of them an image bank in one form or another. An image bank is essentially a catalog of images that is easily searchable and useable by instructors and students ultimately showing an image on its own page. The term “image” in this context is actually comprised of several elements:

  • The picture itself
  • Callout (Figure 1-1, Figure 1-2, etc.)
  • Caption
  • Credit
  • Relavent metadata, like page number

Image banks come in three forms, two web-based versions—one of which you can see an example of here—and PowerPoint. The PowerPoint-based image bank is literally each slide showing the above components for a given image.

An image bank can have anywhere between 200 to 3,000 images, and not all images have all of those elements. A book will have any or all the different types of image banks, so content must be displayed in each of those consistently. My team receives hundreds of varying types of content requests each month, but the image banks are the biggest project in terms of scale and scope by a huge margin. We do fewer image banks than we do of other requests, but image banks take the longest and are the most time consuming.

A long time ago, we used to make image banks by hand. The process took months and was riddled with errors since someone manually proofed and placed at least 1,000 content and data points. Then, a former employee wrote an Applescript and everything changed.

Enter Applescript

Applescript has helped us make the process much more reliable and faster. Essentially, the way the creation process works is the following:

  • My team collects all content into an Excel spreadsheet templates (this is still done manually, though XML is going to change that really soon)
  • Editorial edits the content to meet common and lowest-common-denominator image bank specs
  • My team takes the edited spreadsheets and runs a script to create the product.
  • The script extract the content from Excel, massage as required, and then transform into the required format
  • The resulting files are returned for proofing and any changes made are done in the spreadsheet (they are almost always typos) and the process starts at the beginning again

The “resulting files” can be made in any number of ways, but Adobe Photoshop is a common tool because all art must be converted to meet web specs or be usable in PowerPoint. Adobe InDesign is another tool in the process because one of the image banks uses some fairly complex layouts.

The Applescripts do a lot for us in the process:

  • Captions and credits are made “web ready” by converting special characters to HTML entities and removing any odd formatting that may have been carried over from the page layouts.
  • Common text styling is captured from Excel and applied in the output files via a simple spec.
  • Layouts are created and optimized dynamically via simple rules with no manual intervention.
  • Files are appropriately named, errors logged, and so on.

Applescript makes inter-application automation a breeze. Most scripts go between the Finder, Excel, and then InDesign or Powerpoint at the click on icon, and I have developed scripts and processes that I can hand off to people with no programming knowledge of any kind. These scripts are anywhere between 1,200-3,000 lines of code; they are major applications unto themselves. My team doesn’t even open the script in Script Editor or Script Debugger to run it. I always felt the real power of Applescript was the ability to create complex workflows so easily, but that’s provided application developers are willing to support Applescript. In this case, Applescript saved the day and continued to for a long time.

Enter JSX, Wither Applescript

Adobe wants to own their entire automation stack. Since their introduction of the ExtendScript ToolKit using their version of Javascript as the basis (the file extension is “jsx” hence what I am calling it here), Adobe’s Applescript support has deteriorated to the point where it is now useless to us.

The first nail in the coffin was when Photoshop could no longer open a file via Applescript. But with the release of Creative Cloud and the associated updates to Creative Suite 6, now their dictionaries are stripped of the needed command to create a hybrid Applescript-JSX workflow. The final nail that came with Creative Cloud is that I have since discovered that the JSX framework is no longer accepting arguments as part of script calls from external commands. Everything just passes through as undefined. This may be a big bug, but given the (very) niche nature of my environment, I don’t hold any hope this will be fixed.

I have been porting most of my Adobe automation code to JSX—rebuilding much of what I have done in Applescript in JSX—an effort that has taken better part of this year to get ahead of these changes. I had originally banked on Scripting Bridge to help with pushing content from Excel to the Creative Suite. My goal was to consolidate all of my significant workflows into a Cocoa application for ease of use and better long-term project management. But since that relies on the same frameworks as Applescript—see Apple’s Open Scripting Architecture that Adobe is clearly looking to drop—I now have to look into alternatives.

Alternatives

Again, from the article:

It wouldn’t be so bad were there an alternative.
iWork 13 — A Huge Regression

Alternatives to Applescript depends entirely on one’s own technology stack and a willingness to accept change. To put it another way, alternatives to Applescript simply do not exist only when one decides to stick with Mac OS.

I am exploring options in still maintaining consistent, reliable workflows to create image banks with Adobe Creative Cloud on the Mac. Right now, I am focused on the idea that since I am working with as much text as I am art, I can leverage tab-delimited files (a perennial favorite around here) saved from Excel and JSX’s baked-in XML parsing to create an content transfer spec by creating an in-house content transfer file specification. This means that one-button-click processes turns into n-button-clicks processes, but I still get to keep my investments in Mac OS. I’m not thrilled with this and neither will my team.

Another option is to remove Creative Cloud from the process completely, especially since we are moving everything to web anyway and layouts can be managed with CSS. The PowerPoints are a sticky wicket since that is one of the most-requested formats, and we are beholden to our customers. If not, I would jump to PDF in a second, making the slides using LaTeX. Convincing Sales and Editorial to migrate to PDF, however, would be a huge leap, but one that I am also considering for a variety of reasons, not just this.

Luckily for me, Adobe’s applications are cross-platform as is their JSX platform, and Adobe still maintains a Visual Basic API. The implication is that when Microsoft decides to drop Applescript support from Office as well—given that Adobe and Apple are moving in this direction, why should Microsoft continue?—I can make the move to Windows for process automation and use Visual Basic to automate Excel and then, with some relatively minor changes, carry over my JSX code as writ.

There are some serious caveats to significant of a migration, training and monetary investment notwithstanding. Primarily, if I can reliably call a JSX script with arguments via Visual Basic, then I have effectively re-established where I was before Creative Cloud was released. If Adobe has not treated Visual Basic with the same disdain as it has Applescript, then I could eventually be back where I was before JSX was released. That will take time, but in talking with our preferred vendors, the fact that we are using Macs for such large-scale production makes us the odd one out anyway. I will always have a MacBook Pro by my side for everything else, but then I lose my ability to effectively replicate my environment across both computers.

Switching platforms is something that I don’t want to do but the enormity of my Applescript code cannot be overlooked. At the same time, I have made significant investments in Mac-only applications and technologies—BBEdit, OmniGraffle, Cocoa framework—and now Unix now that I have been in school these past couple of years. There is still a lot of Applescript code running around here (and elsewhere) that is useful and viable.

But, when I do take a step back and survey the current environment as it pertains to publishing automation, there is little being done these days that is specific to Mac OS, and with the limited resources available to me I need to consider where I will receive the most support from the platforms upon which I am supporting my career. I’ll try a couple things first and then weigh my options.