Feeds:
Posts
Comments

Archive for the ‘Software Development’ Category

It had been a while, many years actually, since I needed the Windows equivalent of “touch“. You don’t know that command? All it does is change to modification date and time of a file (or a series of files) to the current date and time of the computer. I used to turn to the Cygwin toolkit to get things done, in the days when corporate Windows PC’s weren’t so closed off and you could install your own tools.

Luckily for me there is an equivalent in Windows, on the command line. You can use this somewhat strange command to get the same result:

copy /b filename.ext +,,

Yes, that’s a plus sign followed by two commas at the end. I’m writing it up here because I know I won’t remember that correctly in a few days!

Read Full Post »

Charlie Arehart is a well-known ColdFusion specialist. Two days ago, he wrote a blog post explaining why one should be careful about securing ColdFusion Archive (CAR) files. The Adobe ColdFusion team isn’t very explicit about the issue, telling us in small print that we should delete those files after using them – but does not explain why we should do so. So Charlie explains it in great detail – if you work with Adobe ColdFusion, you should check out his blog.

Now Charlie only mentions versions 2016 and 2018 of ColdFusion, and I know that there are still developers around that work with older versions – actually, I’m one of those: ColdFusion 11 is what I support (and occasionally develop for) since 2015. I have been using  .car files for installing CF servers, and I had already been looking at what they contained. But I had never seen the ‘seed’ and ‘algorithm’ strings Charlie writes about, but I could have overlooked them. So I went in again today, to verify things.

I can confirm that .car files created in CF11 do NOT contain those strings. But before you start celebrating, I must warn you that this probably means that the situation is even worse than for more recent versions. Because CF11 will write (encrypted) passwords into a .car file, and yes: those files can be used to reconfigure another server, passwords included ! Which probably means that all CF11 runtimes use the same seed and algorithm, rendering  CF11.car files containing passwords even more insecure than later versions

I did not know about all this until yesterday, but I seem to have circumvented the problem: I wrote an application to install datasource definitions on the servers rather than use CAR files. That offers multiple advantages: the code  (and hence the definitions) is under version control, and can only be accessed by authorized users; we have different definitions for different environments; etc. And the.car files I do use have no passwords in them – whew!

But it’s clear that it pays take this issue into account as a ColdFusion developer or administrator, whatever solution you choose (and Charlie has a few propositions).

Read Full Post »

Strong words, but there’s more than a grain of truth in them: “Why Kubernetes is The New Application Server“. “Classic” application servers like those for Java are no longer sufficient by themselves to build a platform that can serve big internet-applications with a large, world-wide audience. And in the world of “containers” Kubernetes seems to be king, as far as I can tell.

Container ship at sea

(Photo by GPA Photo Archive – Original on Flickr)

In order for containerisation to work, applications must be properly “documented” – in fact, the bulk of the “configuration documentation” will somehow be part of what is needed to get those containers up and running. Around the time I read up on Kubernetes I stumbled onto something called “The Twelve-Factor App” – can’t remember who pointed me there. This methodology (it’s not an app!) describes a well-documented way to build, configure and run a cloud application – a laudable objective.

At work, we have tried to describe our applications in order to migrate them to another (Windows) domain with new (better) rules about access control, database access, etc. But things aren’t working out as they should. We do have documentation, although I’m not sure how useful it is outside of the context of passing relevant information from the developers to an external partner that will implement parts of the configuration. Additionally, we have described lots of “what“s, but almost no “why“s – which might be essential in the coming months and years as the applications continue to evolve…

Ideally, I would have loved to have a decent ‘methodology’ for documenting application essentials when we were building our applications. Trying to figure out what has to be done to get things up and running again on new servers has become something of a nightmare. That is even more so when the application you’re handling was developed by someone who’s no longer available for questioning!

The Twelve-Factor app may turn out to be very useful, although I suspect it is incomplete. I don’t think there is a single method for completely describing and documenting applications and systems that extend beyond the most simple cases. Any ‘methodology’ to build software is bound to need more or less tweaking to fit your (or your company’s) way of working. Getting to know methodologies other than the one you’re using is a good way of discovering what you need to get better!

Read Full Post »

My setup has been the same since quite a few years now: I have a Keepass file on Dropbox, and I use several different applications and apps on multiple devices to access and update that file. Which applications, you ask?

On my Macs as well as on my Xubuntu machines I will use Keeweb. Despite its name, it gives you a desktop application that natively accesses (and syncs) files on Dropbox. This is the application I go to for when I want or need to reorganise the Keepass file, e.g. to rearrange groups or import lots of account data.

I would use Keeweb on a Windows PC as well – if I had one. At work, we have no free choice of which application to use to store passwords, but luckily we do have the “official” Keepass Password Safe at our disposal.

On Android my favourite Keepass app is called Keepass2Android. I will admit that I made that choice a few years ago, and haven’t checked on its competitors recently (are there competitors of note, by the way?). But it does what I need it to do; it accepts Dropbox as cloud storage and it will even merge changes from the local version and the Dropbox version when it detects differences between the two during the synchronisation process. That last one is a killer feature, and it hasn’t failed me a single time in the years I have been using it.

On iOS the situation is a little more complicated – at least, that how it feels to me. I wrote earlier about KeePassium, and that is still my app of choice. I like the interface, and it does all I need when I look for account info (you can store more than just passwords there!).

But in order to sync my central file on Dropbox, on iOS the app has to go through the “Files” app from Apple. Files-the-app is capable of showing files of all kinds on the iOS device, as well as the files on several cloud file systems, like Dropbox. What is less clear to me, however, is how quickly “Files” notices changes on Dropbox and picks up the latest version of my central KeePass file. I also have had trouble getting the latest version of my file (as changed on Android, for example) onto my iPhone. Although I must admit that the last few weeks fared better: I haven’t noticed anymore missing syncs lately. What I can’t say is whether the issue was/is with Files rather than KeePassium or even my internet connection…

Anyway, when it comes to passwords I want to be sure that I’m not missing any information – or worse: I don’t want to overwrite my updated central file with an older version on iPhone! That’s why I currently always check the “last updated on” date of my Dropbox file in Files before opening the file again. Of course my Dropbox account is protected with a password, but I don’t think that is what Andrei Popleteev means when he’s writing about “How to sync KeePassium with Dropbox“.

Manually checking the file date on iOS is not an ideal situation, I know, but to me that check is a small price to pay for the greater good of having my account data available on all the platforms I use! And for me, KeePassium is still the way to go on iOS.

Read Full Post »

Contrary to most pure hardware tools like a hammer, software tends to evolve over time. These days, software evolves faster than ever before – and at the same time most pieces of software that we use regularly are also interconnected with other software. Think of your smartphone, where the operating system updates the apps running on the device, while some – if not most – of the apps require connections to other infrastructural software and “platforms” from the likes of Google, Apple, and many others. Synchronising account and application data is getting more important every day, the more so now that more and more people have more than one device. No wonder then than sometimes things take a turn for the worst…

Case number one: I have been using a couple of home-brewed scripts to get the daily production numbers of our solar panels from the SMA monitor to an Xubuntu computer, and then transfer them to a Google Drive for storage. I used Grive2 to sync new or renewed files to Google Drive, until that failed as I reported on December 15th. Google started restricting OAuth access rights in November 2019, and that poses a problem for tools like Grive2.

My replacement solution using Jdrivesync is actually victim of the same OAuth change, although it is less evident: it can still add files to Drive but fails when reading the metadata of Drive files (and hence is incapable of replacing them as well).

Today I took the time to tackle the issue head-on, and started by re-reading the instructions on Grive2. That answered my question of a few months ago: I now know why Google changed its approach. The Grive2 site also explains how to circumvent the limitations, by creating your own Google API project and OAuth credentials. It’s not the fault of the Grive2 author, but man oh man, what a convoluted process is that. You get to answer a pleiad of questions that may be easy to understand for a seasoned Google developer, but not for an end user trying to get a simple sync script to work again! In the end, after a series of dire warnings by Google during the process, things started working again. Which is nice. But I’m still not sure for how long this will continue to work. That’s not reassuring for a solution that is supposed to work without a hitch for at least 10 more years or so.

I think the burden here is on Google: it would be nice if they could figure out a way for single end users to get a single application instance (project) up and running on a single account in an understandable process. Because that is what I needed: a way to tell Google that MY Grive2 script will sync MY data from MY computer to MY Google Drive. A simple process does not need to bother me with questions about GSuite domains, privacy declarations, consent screens, and what more. Please, Google?

Case number two: since a few weeks I’m a happy user of KeePassium. I use it on my iPhone as well as on an iPad, where both devices open the same KDBX file. Since I also still have an Android device running Keepass2Android, I store the KDBX file in DropBox. This setup seemed to work OK, until a few days ago when a new account added on the iPad did NOT show up on the iPhone nor in Keepass2Android. After a few tests and trials I ended up with saving the file explicitly to DropBox and reopening it on both the iOS devices, and later synced Keepass2Android as well. The latest changes in the file are now visible on all three machines, so that’s good.

However, I fear that I may have lost one earlier password change. I’m not in any position to blame either DropBox, Apple’s Files app, or KeePassium, since I cannot (yet?) explain what happened. So while the situation is “under (manual) control” now, I keep wondering what will happen when I apply the next changes to the KBDX file. Here, like in the case above, the synchronisation should ideally happen without any special interaction on my part. Unfortunately, as long as I’m not certain that the complete setup works “as expected” I may as well continue to sync by hand – and that is exactly what smart software is supposed to automate, no?

Conclusion? As a developer of sorts, I’m familiar with all aspects of software, good and bad alike. I know things can go awry, and I know how to try and figure out what goes wrong and how to try and resolve the issue. But I’m part of a minority, speaking globally, and I can imagine that many (most) people would just declare defeat and call the software they were using “buggy” or “bad” or “useless”. While that may true in some cases, it mostly shows that developers and publishers of software will need to take more care when building their products: no software is an island, and many if not all software tools will have to talk to others – hopefully in a polite and productive manner. Not an easy task, but possibly essential if the tool has to be around for a long time.

Read Full Post »

“Seeing” things as colors or sounds has always intrigued me, so I had to have a look at the “What Color Is Your Name?” website. Don’t expect an extensive and scientific explanation of the phenomenon; just enjoy the results. Here’s what the alphabet look s like for Bernadette:

I can see this site being used to select a color scheme by website designers!

Read Full Post »

A good month ago, I had to switch from Grive2 to Jdrivesync on my little Xubuntu machine, because Google doesn’t like the former software. Unfortunately, Jdrivesync is not without problems.

The biggest issue is that Jdrivesync is not capable of updating an existing file in de Google Drive with a fresher version from my machine. And it turns out that I’m not the only one (nor the first) one to experience this error, as detailed in this Github error report called “Error if updating a remote file“.

I’m the first one to admit that software without bugs is very, very, very rare ;-)

But a bug report without response in more than 20 months is a clear sign of abandoned software. So I’m looking for another solution – suggestions are more than welcome (I’m not in a position to start learning the ins and outs of the Drive API to see if I can find the cause of the problem).

Read Full Post »

In “The State of the Octoverse ” for 2019 Github writes:

And for the first time, Python outranked Java as the second most popular language on GitHub by repository contributors.

Not bad – JavaScript takes first place, Python and Java follow.

Talking about Python allows me to announce that I hacked my first Python application ;-)

You see, I’m still running an old version of Trac to track issues and document my developments. Old means: version 0.10.5, and the problem is that I don’t have the time at this moment to upgrade to the latest version. So when I had to move to a brand-new Windows 2016 server I just took the existing installation instructions and got Trac running fine, although not under the wings of IIS. Except for one thing: the new server was behind a firewall and a virtual IP address (VIP), the latter also taking care of HTTPS to HTTP conversion. So when my browser sends an HTTPS request, Trac sees HTTP traffic and responds with HTTP redirects when replying to certain actions. Those replies turn into error pages in the browser, since HTTP traffic from the browser does not pass the firewall and is not intercepted by the VIP…

I have been reading up on Python, but I can’t say I have developed anything seriously in it. Python is however sufficiently readable so that I easily found my way in the code. To solve my problem, I just had to force Trac to always redirect to HTTPS URLs. As far as I can tell, I had to change but a single line of code to make that happen. Here’s the code of the ‘_reconstruct_url’ function after my hack:

Image of the Python code for the function '_reconstruct_url'

I changed lines 226-227 to the single statement in line 228

I’m calling this a hack because it only solves my problem; this is not something you want to submit to a public repo. It also means that the application no longer responds to standard HTTP requests. But it’s good enough for now.

From what I have seen on the Trac website there are solutions for running later versions of Trac over SSL, but most (all?) of them seem to rely on an SSL front on the same server as Trac. Too bad that such a solution won’t work for me. Guess I’ll have to study Python and Trac a lot deeper to find the best way to cope with the architecture at work.

Read Full Post »

So you’re developing a ColdFusion application and you need to access the output of a Windows program, like ‘netstat.exe‘ to take the example from the Adobe documentation?

The documentation on how to do that is relatively complete, but fails to emphasise an important point. Try this, and you’ll see… nothing, no output:

<cfexecute name="C:/Windows..../netstat.exe" variable="output" />
<cfdump var="#output#" />

The important point is that you really need to specify a non-zero ‘timeout‘ argument if want to capture the output of the Windows tool in a variable. Simple, but yes: this one had me stumped for 90 seconds!

<cfexecute name="C:/Windows..../netstat.exe" variable="output" timeout="1" >
<cfdump var="#output#" />

If you forget that ‘timeout‘ argument the command you issue will execute, but you won’t know its output, since ColdFusion is already further along in its execution. You may have to experiment a bit with the value for the timeout, depending on the specifics of the machine you’re running ColdFusion on and the execution time of the command called (obviously). Not specifying the timeout will launch the command, but it won’t wait for the command to end – which can be useful as well – but not always.

Read Full Post »

Since many years I have been writing down the guidelines I use when developing software. Mainly because it’s easy to forget some of them, but also because at times you need to explicitly ponder their weight in a trade-off between two or more “principles”. One thing is clear after all those years I have spent writing and reading software, whether it was written by me or someone else: writing good software and writing good code is difficult.

Michael Foord has written up his “30 best practices for software development and testing“, and it’s hard to disagree with his list: there is a lot of good advice in it! If you’re just starting to code, you can’t hope to apply all 30 “rules” at once. The only gripe I have is with the title of his post: in my view, it should have been published as “30 best practices of software coding and testing“. There is, after all, a lot more to software development than coding.

Let me prove that statement. For heaps of excellent advice on the subject of software development in general I always pick up Alan M. Davis’s “201 principles of software development“.

My copy was printed in 1995, and it seems there are no reprints – which is a shame, because Alan Davis did a great job of consolidating proven principles in categories ranging from requirements analysis over design to coding and testing and product assurance. Each principle is explained, and Davis adds at least one reference text for almost every principle. In summary, the content of this book is not very original but it remains a valid and comprehensive overview of good software engineering practices. That remains true, even after more than 20 years!

PS. I already wrote earlier posts on the subject of software engineering principles, e.g. in May 2015

Read Full Post »

I encountered this weird situation last week: while testing ColdFusion code from a previous decade, I would see the output of part of the screen twice. Just like that – I had not changed a thing to the logic of the code, just cleaned up a bit of what I consider to be “bad” formatting in code written by others a long, long time ago (2005 or so).

I spent hours of debugging the code, even invited a few colleagues to help me – and then suddenly it struck me: CFML may look like XML, but it isn’t XML. Some CFML tags do their thing a second time when the tag is explicitly closed. One of those tag is <cfmodule>. I never use it myself – I prefer components and functions. But the code under scrutiny included a <cfmodule template='whatever.cfm >, and to “clean things up” I had added a slash at the end, like this: <cfmodule template='whatever.cfm />. Hence the double execution. So don’t do that, will you? Avoid closing <cfmodule>!

You can imagine my state of mind when I realised my error, don’t you? Yeah… I had seen this problem before, but much too long ago to actively avoid it in practice.

Before lecturing me on the subject: I know, I should be writing cfscript… but I don’t have the luxury to rewrite code that is more than 10 years old. So it will stay as it is.

Also: I consider this to be a bug in Adobe ColdFusion (at least in versions 6, 8 and 11). There’s no logical reason for such behaviour. I don’t know if Lucee or BlueDragon act similarly, but I hope they’re smarter than that!

And for the record: for me, code is “badly” formatted when it is hard to read – by me. I know compilers and computers have no trouble handling things like whole programs on a single line, or lacking coherent indents (unless it’s Python, of course), or unclosed XML tags (elements), to name but a few examples. I want code to easily readable by humans, and certainly by me – especially if I’m responsible for its maintenance…

Read Full Post »

Sure. As told on the Zappa webpage:

Zappa makes it super easy to build and deploy server-less, event-driven Python applications (including, but not limited to, WSGI web apps) on AWS Lambda + API Gateway. Think of it as “serverless” web hosting for your Python apps. That means infinite scaling, zero downtime, zero maintenance – and at a fraction of the cost of your current deployments!

That’s less revolutionary than Frank Zappa, of course. But it might come in handy, once I find the time to do some real Python development… Then again, you will probably beat me to it!

Read Full Post »

We’re six years later, and I still haven’t gotten around to any kind of “Tinkering With The Raspberry Pi“. That does not mean that I still have to write down the production numbers from our solar panels by hand, however. The Asus eeePC, running Xubuntu and a bit of software a former colleague of mine and I hacked together, takes care of that. In doing so, it constructs a number of text files: one for each day, listing the current production in Wh every ten minutes, and one for each year, detailing the total production for each day. The backup of these files is made every day by a tool called ‘grive2‘ (but I’ll write about that later).

The setup works fine, almost all the time. But somehow the SMA Sunny Boy gets confused and creates ‘yearly’ files for years other than the current calendar year. Those files are utterly useless and clutter the hard disk as well as the backup, so I decided to get rid of them automatically. To prepare for the first days of a new year, the script should also be able to leave the file for the previous year in place – there may be two valid ‘yearly’ files in January, should I fail to archive the old year on New Year’s eve or on Jan. 1st.

To exercise my *nix shell skills, I decided to do that in ‘bash‘ rather than extend the current Python tools.

As is my habit, I decided to start with a demo script that does what I want on dummy data. For demo purposes the JDoodle website is a great resource, at least for ‘bash’ scripting (I did not try any of the 67 other languages available on the site). This allowed me to work on the code on my Mac-with-big-screen, and take the necessary screenshots for this post.

Here is the code I came up with:

Click on the image to get it in the form of a PDF file,
ready for copy/paste operations.

Nothing spectacular, as shown by the output. Now all I have to do is turn this into a little non-demo script and add it my crontab on the eeePC… Come and see in six years or so ;-)

PS. I’m just dabbling in bash scripting, so if there are better solutions for my problem, don’t hesitate to explain them to me, please.

Read Full Post »

The SD Times website writes up the state of affairs on Free and/or Open Source Software: Open source at 20: The ubiquity of shared code. Even after 20 (or more) years, the situation isn’t always clear, certainly not for new developers. So this article is a good start if you’re new to software development.

In the year 2000 I compiled on the Free Software page in this site. I’m pleased to see that the texts and sites on the page are still relevant. Only two sites disappeared (linuxppc.org and opensourceit.com); the rest is still thriving and relevant! Well, except for the link to Reddit – still a remarkable site, but no longer just for FOSS fans.

A landmark paper about Free and Open Source Software

Read Full Post »

The BoingBoing website pointed me to the Programmer’s Oath. Good initiative, and I do agree with every one of the items.

As usual, of course, my mind started analysing the text, and soon concluded that 8 of the 10 tenets are not specific to programming, but could be applied to any profession! And tenets 2 and 6 don’t need big changes to make them more generally applicable. So what user Widdershin came up with is the base for moral behaviour that all humans could/should fulfill.

Well, being cynical at times just like anyone, I should perhaps exclude politicians…

Read Full Post »

Older Posts »