Sunday, January 18, 2015

Call for help with open source project "CLPM"



CLPM is my “Command Line Project Manager”. It’s a tool I wrote and have been using myself for several years now, and I am releasing it in the hope that others might find it useful.

Also, if you have been looking for an open source project to contribute to, here’s your chance! I don’t care what your level of experience is, if you think you have a useful comment or contribution, I’d like to hear from you!

There is a Todo section in the README, but I want to add a couple notes here:
  1. It’s currently not packaged, nor does it have an installer. This probably makes it much less likely to be adopted.
  2. I’m not sure how to promote it to make sure its audience (developers/sysadmins maybe) at least get a chance to see it, even if it ends up that it’s useful to nobody but me.
The project page has more details: https://github.com/tinypigdotcom/clpm

Thursday, January 08, 2015

CPAN Pull Request Challenge: The Pull Request

In my last report, I had received an ok from the module’s author to proceed with a pull request. The first thing I did, as I had with the modules referenced by Devel::StackTrace::WithLexicals, is to write an example for myself so I could see how it worked. Working from the synopsis, I had a difficult time writing a working piece of code. It occurred to me that this was the very place I could start with a Pull Request.

Since the module’s functionality combines that of two other modules (Devel::StackTrace and PadWalker), I wrote a working example showing both sets of functionality: specifically, the ability to walk back through the stack and the ability to read and modify a previous frame’s lexical variables.

Since I had not yet specified my area of focus to the author, I send him another email including the synopsis I had written and asked him for his thoughts on my changes. He liked them but asked that I replace my foos and bars with more realistic names. I had thought that foo and bar kept out of the way so that only the module’s functionality was on display, but he’s right: although the fake variable names do stay out of the way, they provide zero context for the user to become oriented while going through the code. I decided to use a problem domain that should be familiar to many developers: the shopping cart. Here’s the synopsis I ended up with:

use Devel::StackTrace::WithLexicals;
sub process_user {
    my $item_count = 20;
    price_items();
    print "$item_count\n";    # prints 21
}
sub price_items {
    my $trace = Devel::StackTrace::WithLexicals->new(
        unsafe_ref_capture => 1    # warning: can cause memory leak
    );
    while ( my $frame = $trace->next_frame() ) {
        my $item_count_ref = $frame->lexical('$item_count');
        ${$item_count_ref}++ if ref $item_count_ref eq 'SCALAR';
    }
}
process_user();

I followed this excellent step-by-step guide on submitting a Pull Request and just a little while later it was approved!

Afterward I found this well-written post about having a good synopsis. What I had written conformed pretty well with the suggestions.

I had never done a pull request before this. There have been a couple times I contacted authors in the past to point out a bug or add a function, but most of the time I have not been an active participant.

Why?

  1. I am often busy with my own projects. Lately, I have tried to get more involved in the community. Much of this inspiration has come from the company I work for now, OmniTI, whose philosophy is very pro-open-source, speaking, contributing, etc. (side note - we’re hiring!)

  2. I can still be painfully shy. It sounds childish to put it in those terms, but it’s also accurate. It is difficult for me to “put myself out there” sometimes even if just in the form of an email.

  3. I had no idea where to start. This sounds like a cop-out, and in some ways it is, but I can honestly say I’ve spent hours looking around at different open source projects and not really found any place it looked like I could help.

But my shortcomings illustrate some of the things that make the CPAN Pull Request Challenge such a great project.

  1. It’s got a time-sensitive feel to it. It has dates attached to it, so feels like something you need to focus on now.

  2. It gave me an authority to point to when reaching out. Author: I’m contacting you not because I personally felt there was anything wrong with your module, but rather because this challenge told me to. So don’t be mad at me! :)

  3. It gave me a starting point. Being assigned a module is much more focused than having someone just say “Hey - contribute to something!” Now I could still have been bogged down in “what exactly do I contribute to in this module” but the challenge comes with tons of ideas on things to look at as well. Failing that, there’s always “Kwalitee”.

So, thank you, Neil Bowers - both for being a positive influence on the Perl and open source communities, and for helping me get out of my shell a bit.

Friday, January 02, 2015

My First Day on the CPAN Pull Request Challenge


So yesterday, I started on the CPAN Pull Request Challenge. I was assigned the module Devel::StackTrace::WithLexicals, so, I read the doc and installed it.

The doc references two other modules’ influence: Devel::StackTrace and PadWalker

So I figured I’d have a look at those first. Combining two examples from the doc for Devel::StackTrace, I wrote this:

#!/usr/bin/perl

use strict;
use warnings;

use Devel::StackTrace;

sub foo {
    bar(@_,1234);
}

sub bar {
    my $trace = Devel::StackTrace->new();
    print $trace->as_string(); # like carp
}

foo('bing');

Which gave me this:

$ ./d.pl
Trace begun at d.pl line 13
main::bar('bing', 1234) called at d.pl line 9
main::foo('bing') called at d.pl line 17

Neat!

Next I looked at the doc for PadWalker. Based on the example provided, I wrote:

#!/usr/bin/perl

use strict;
use warnings;

use PadWalker qw(peek_my);

sub foo {
    my $abc = 123;

    bar();

    print "magic! $abc\n";
}

sub bar {
    my $h = peek_my(1);
    ${$h->{'$abc'}}++;
}

foo();

Which yields:

$ ./e.pl
magic! 124

So bar() increments foo()’s lexical variable $abc. What kind of witchcraft is this?!?

These are both very interesting modules. Changing lexicals from another scope defeats the purpose of a lexical, but as the doc points out, it can be useful for debugging purposes.

I reviewed the documentation for Devel::StackTrace::WithLexicals and the code, both of which can be viewed online at meta::cpan. In the left column are the links “Source (raw)” and “Browse (raw)” which are both handy to use when viewing the source.

Nothing immediately jumped out at me after reviewing the pull request ideas so I decided to contact the author to see if he had anything specific he’d like to see done. The author’s contact info is usually provided in the Author section of the documentation.

This is the email I sent:

Hello!

I have volunteered in the CPAN Pull Request Challenge
and have been assigned your module Devel::StackTrace::WithLexicals to
contribute to if possible.

Ideas include, but aren't limited to:
http://neilb.org/2014/12/31/pr-ideas.html

I'd like to know if you are receptive to contributions to this module and if
you had any particular needs or wishes.  I am willing to contribute with any
aspect (features, bugs, testing, documentation, etc.)  If you are ok with
contributions (pending your approval of course) but you don't have anything
specific in mind, I will review the PR Ideas list myself and see if I can find
a way to contribute.

Thank you,
David M. Bradford
http://tinypig.com

This morning I received a positive response from the author. Time to get to work!

Sunday, December 21, 2014

Vim Macros: A More Complicated Problem

Here’s a more difficult problem than my previous post on Vim macros: I collected a bunch of links to post on my website using a simple 2-column table:

blogs.perl.org  |http://blogs.perl.org/mt/mt-cp.fcgi?__mode=view&id=3159
Wiki            |http://tinypig.pbworks.com/
Tumblr          |http://tinypigdotcom.tumblr.com/
WordPress       |http://tinypig.wordpress.com/
Perl Monks      |http://www.perlmonks.org/?node=tinypig
GitHub          |https://github.com/tinypigdotcom

My HTML links template I want to add them to is basically this:

<h3>Links</h3>
<ul>
<li><a href="">LINK</a><br/></li>
</ul>

Appending the first file to the second, I get this:

<h3>Links</h3>
<ul>
<li><a href="">LINK</a><br/></li>
</ul>

blogs.perl.org  |http://blogs.perl.org/mt/mt-cp.fcgi?__mode=view&id=3159
Wiki            |http://tinypig.pbworks.com/
Tumblr          |http://tinypigdotcom.tumblr.com/
WordPress       |http://tinypig.wordpress.com/
Perl Monks      |http://www.perlmonks.org/?node=tinypig
GitHub          |https://github.com/tinypigdotcom

Note the pipes (|) separating the columns. Having these (or some other unique character) make it much easier to work with the data because we can use Vim’s F find command to move to the exact location we need. Also note there should be a blank line at the end of the file so that we can go down successfully and our find on the pipe symbol will fail, ending the recursive macro.


Step Keys Meaning
1 qqq Start and immediately stop recording macro into buffer ‘q’, to empty it
2 qq Start recording a macro into buffer ‘q’
3 0 Go to the beginning of the line
4 f| Find the next pipe symbol
5 0 Go to the beginning of the line
6 3k Go up three times
7 Y “Yank” the current line (in this case, the link template line) into the buffer
8 p Immediately “put” the line right below, effectively duplicating it
9 3j Go down three times
10 dt| “Delete to” pipe, which also captures the deleted text into the buffer
11 4k Go up four times
12 f>f> Find greater-than symbol, then find the next one
13 p “Put” the buffer (the data from the first column we deleted 3 steps back)
14 ge “Go” to the “end” of the previous word
15 l Go right
16 dt< Delete to greater-than symbol, effectively removing trailing space
17 0 Go to the beginning of the line
18 4j Go down four times
19 l Go right, skipping over the pipe symbol
20 D Delete the rest of the line (again capturing the text)
21 4k Go up four times
22 f" Find next quote symbol in the current line
23 p “Put” the buffer (data from the second column)
24 0 Go to the beginning of the line
25 4j Go down four times
26 dd Delete the current line
27 @q Add the instruction to start running macro ‘q’ again, creating the recursive call
28 q Stop recording
29 @q Actually run macro ‘q’



Here’s what it looks like in practice:

Wednesday, December 17, 2014

Fixing Multiple Lines with Vim Macros


Sometimes I have a tedius code change where I have to perform the same task on multiple lines in vim and I can’t use ‘.’ because each line is a little different or the changes are too complicated. In this situation, I use a recursive macro in vim.

Here, I started with a list of items that I want to turn into a dispatch table - the code itself isn’t important here, it’s the transformation that I want to illustrate:

my %actions = (
    'start_shopping',
    'add_item',
    'remove_item',
    'use_coupon',
    'checkout',
);

Step Keys Meaning
1 qqq Start and immediately stop recording macro into buffer ‘q’, to empty it
2 qq Start recording a macro into buffer ‘q’
3 0 Go to the beginning of the line (always start there, if possible, for consistency)
4 f' Find the next single quote (always start with a find. That way, in the infinite loop we are creating, when it fails the loop will end)
5 l go right to get to first “real” character
6 yw Yank the word
7 $ go to end of line
8 P Put (before the comma)
9 F' Find the previous single quote
10 x delete it
11 i start inserting
12 <SPACE>=><SPACE>\& (literal characters to insert)
13 <ESC> Escape, leaving insert mode
14 F' Find the previous
15 x delete it
16 0 Go to the beginning of the line
17 j Go down a line
18 @q Add the instruction to start running macro ‘q’ again, creating the recursive call
19 q Stop recording
20 @q Actually run macro ‘q’

If you have made a mistake and it starts going haywire (this happens to me once in a while) just <CTRL>-c and <ESCAPE> and u for undo and it should go back to the way it was before you ran the macro.

Here’s what it looks like in practice:


Note: I have this line in my .vimrc file:

map z @qz

Since I don’t use z’s default functionality, instead I have it run @q on an infinite loop which allows me to skip step 18 from the example and then step 20 just becomes “z”.

For this next group of commands, I’m going to use the same concept to align all the =>. The block must begin with the line where => is furthest to the right. If it doesn’t belong there, you can always put it there to align it and then move it back to its place afterward. We’ll start on the second line this time since the first line is already positioned correctly.

Step Keys Meaning
1 qqq Start and immediately stop recording macro into buffer ‘q’, to empty it
2 qq Start recording a macro into buffer ‘q’
3 0 Go to the beginning of the line
4 f= Find the next =
5 50i<SPACE><ESC> Insert 50 spaces to make sure it’s long enough
6 0 Go to the beginning of the line
7 k Go up
8 f= Find the next =
9 j Go down
10 dw “Delete word” in this case will delete to the next non-space
11 0 Go to the beginning of the line
12 j Go down a line
13 @q Add the instruction to start running macro ‘q’ again, creating the recursive call
14 q Stop recording
15 @q Actually run macro ‘q’

Here’s what it looks like in practice:


Credit to Hongleong for the qqq emptying idea (there is much more info at that link regarding recursive vim macros)

For routinely aligning text in vim, I found this video to be extremely helpful.

Just in case that disappears, I have copied two of the links included there: the Tabular plugin and a macro which uses it.

Animated GIFs created with LICEcap

Friday, December 05, 2014

Using Files for Inter-process Communication



I was abusing a test suite I had written to add test data to the database and I was running it inside an infinite loop because it was going to take a while to generate the 10,000 records I wanted. When it came time to stop, I had difficulty either control-c-ing or killing the right processes to get it to quit, so the second time I ran it, I did this instead:

while true
do
    if [ -f $HOME/stop_while ]; then
        break
    fi
    ./thing_i_wanted_repeated
done

Using files for inter-process communication seems so much more straightforward to me.

Thursday, December 04, 2014

XML Nested CDATA sections




"Nested CDATA sections are not allowed." But I need that! I have XML that contains XHTML which in turn has a CDATA section of its own.  Surprisingly I found the answer at Wikipedia.

Here's my XML:

<apple>
    <banana>
    <![CDATA[
    <script type="text/javascript">
    /* <![CDATA[ */
    var language = "en";
    /* ]]]]><![CDATA[> */
    </script>
    <script type="text/javascript" src="//www.this.com/that.js">
    </script>
    <noscript>
    <div style="display:inline;">
    <img src="//www.this.com/that/?l=b5&amp;scrum=2"/>
    </div>
    </noscript>]]></banana>
</apple>

Although the Wikipedia entry covers it sufficiently, I thought it would be interesting to diagram a bit further (click to enlarge):