see shy jo
The following are the titles of recent articles syndicated from see shy jo
Add this feed to your friends list for news aggregation, or view this feed's syndication information.

LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.

[ << Previous 20 ]
Wednesday, June 10th, 2020
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
7:35 pm
bracketing and async exceptions in haskell

I've been digging into async exceptions in haskell, and getting more and more concerned. In particular, bracket seems to be often used in ways that are not async exception safe. I've found multiple libraries with problems.

Here's an example:

withTempFile a = bracket setup cleanup a
  where
    setup = openTempFile "/tmp" "tmpfile"
    cleanup (name, h) = do
        hClose h
        removeFile name

This looks reasonably good, it makes sure to clean up after itself even when the action throws an exception.

But, in fact that code can leave stale temp files lying around. If the thread receives an async exception when hClose is running, it will be interrupted before the file is removed.

We normally think of bracket as masking exceptions, but it doesn't prevent async exceptions in all cases. See Control.Exception on "interruptible operations", which can receive async exceptions even when other exceptions are masked.

It's a bit surprising, but hClose is such an interruptable operation, because it flushes the write buffer. The only way to know is to read the code.

It can be quite hard to determine if an operation is interruptable, since it can come down to whether it retries a STM transaction, or uses a MVar that is not always full. I've been auditing libraries and I often have to look at code several dependencies away, and even then may not be sure if a library has this problem.

  • process's withCreateProcess could fail to wait on the process, leaving a zombie. Might also leak file descriptors?

  • http-client's withResponse might fail to close a network connection. (If a MVar happened to be empty when it's called.)

    Worth noting that there are plenty of examples of using http-client to eg, race downloading two urls and cancel the slower download. Which is just the kind of use of an async exception that could cause a problem.

  • persistent's withSqlPool and withSqlConn might fail to clean up, when used with persistent-postgresql. (If another thread is using the connection and so a MVar over in postgresql-simple is empty.)

  • concurrent-output has some locking code that is not async exception safe. (My library, so I've fixed part of it, and hope to fix the rest.)

So far, around half of the libraries I've looked at, that use bracket or onException or the like probably have this problem.

What can libraries do?

  • Document whether these things are async exception safe. Or perhaps there should be an expectation that "withFoo" always is, but if so the Haskell comminity has some work ahead of it.

  • Use finally. Good mostly in simple situations; more complicated things would be hard to write this way.

    hClose h `finally` removeFile name
    

  • Use uninterruptibleMask, but it's a big hammer and is often not the right tool for the job. If the operation takes a while to run, the program will not respond to ctrl-c during that time.

  • May be better to run the actions in worker threads, to insulate them from receiving any async exceptions.

    bracketInsulated :: IO a -> (a -> IO b) -> (a -> IO c) -> IO c
    bracketInsulated a b = bracket
      (uninterruptibleMask $ \u -> async (u a) >>= u . wait)
      (\v -> uninterruptibleMask $ \u -> async (u (b v)) >>= u . wait)
    
    (Note use of uninterruptibleMask here in case async itself does an interruptable operation. My first version got that wrong.. This is hard!)

My impression of the state of things now is that you should be very cautious using race or cancel or withAsync or the like, unless the thread is small and easy to audit for these problems. Kind of a shame, since I had wanted to be able to cancel a thread that is big and sprawling and uses all the libraries mentioned above.


This work was sponsored by Jake Vosloo and Graham Spencer on Patreon.

Saturday, April 11th, 2020
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
3:36 pm
lemons

Lemon is one of my things. I homegrow meyer lemons and mostly eat them whole. My mom makes me lemon meringue pie on my birthday. I thought I knew how much work that must be.

Gorgeous whole lemon-meringue pie

Well, that was harder than anticipated, and so worth it. Glad my mom was there on jitsi to give moral support while I hand whisked the egg whites and cursed.

I also got a homemade mask whose quarantimer expired just in time.

But all I really want want for my birthday, this April 11th 2020, is for the coronavirus to have peaked today. I mean, having a pandemic peak on your birthday is sour, but it's better than the alternative.

Please give me that gift. Stay home. Even when some are saying it's over, watch the graphs. Don't go visit even just one person, even on their birthday. I think you can do it.

Yes?

Joey in a rainbow tie-die mask,holding a thumb up.

Lemon-meringue pie slice

Saturday, April 4th, 2020
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
4:56 pm
solar powered waterfall controlled by a GPIO port

This waterfall is beside my yard. When it's running, I know my water tanks are full and the spring is not dry.

Also it's computer controlled, for times when I don't want to hear it. I'll also use the computer control later on to avoid running the pump excessively and wearing it out, and for some safety features like not running when the water is frozen.

This is a whole hillside of pipes, water tanks, pumps, solar panels, all controlled by a GPIO port. Easy enough; the pump controller has a float switch input and the GPIO drives a 4n35 optoisolator to open or close that circuit. Hard part will be burying all the cable to the pump. And then all the landscaping around the waterfall.

There's a bit of lag to turning it on and off. It can take over an hour for it to start flowing, and around half an hour to stop. The water level has to get high enough in the water tanks to overcome some airlocks and complicated hydrodynamic flow stuff. Then when it stops, all that excess water has to drain back down.

Anyway, enjoy my soothing afternoon project and/or massive rube goldberg machine, I certainly am.

Wednesday, April 1st, 2020
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
10:12 am
DIN distractions

My offgrid house has an industrial automation panel.

A row of electrical devices, mounted on ametal rail. Many wires neatly extend from it above and below,disappearing into wire gutters.

I started building this in February, before covid-19 was impacting us here, when lots of mail orders were no big problem, and getting an unusual 3D-printed DIN rail bracket for a SSD was just a couple clicks.

I finished a month later, deep into social isolation and quarentine, scrounging around the house for scrap wire, scavenging screws from unused stuff and cutting them to size, and hoping I would not end up in a "need just one more part that I can't get" situation.

It got rather elaborate, and working on it was often a welcome distraction from the news when I couldn't concentrate on my usual work. I'm posting this now because people sometimes tell me they like hearing about my offfgrid stuff, and perhaps you could use a distraction too.

The panel has my house's computer on it, as well as both AC and DC power distribution, breakers, and switching. Since the house is offgrid, the panel is designed to let every non-essential power drain be turned off, from my offgrid fridge to the 20 terabytes of offline storage to the inverter and satellite dish, the spring pump for my gravity flow water system, and even the power outlet by the kitchen sink.

Saving power is part of why I'm using old-school relays and stuff and not IOT devices, the other reason is of course: IOT devices are horrible dystopian e-waste. I'm taking the utopian Star Trek approach, where I can command "full power to the vacuum cleaner!"

Two circuit boards, connected by numerousribbon cables, and clearly hand-soldered. The smaller board is suspendedabove the larger. An electrical schematic, of moderate complexity.

At the core of the panel, next to the cubietruck arm board, is a custom IO daughterboard. Designed and built by hand to fit into a DIN mount case, it uses every GPIO pin on the cubietruck's main GPIO header. Making this board took 40+ hours, and was about half the project. It got pretty tight in there.

This was my first foray into DIN rail mount, and it really is industrial lego -- a whole universe of parts that all fit together and are immensely flexible. Often priced more than seems reasonable for a little bit of plastic and metal, until you look at the spec sheets and the ratings. (Total cost for my panel was $400.) It's odd that it's not more used outside its niche -- I came of age in the Bay Area, surrounded by rack mount equipment, but no DIN mount equipment. Hacking the hardware in a rack is unusual, but DIN invites hacking.

Admittedly, this is a second system kind of project, replacing some unsightly shelves full of gear and wires everywhere with something kind of overdone. But should be worth it in the long run as new gear gets clipped into place and it evolves for changing needs.

Also, wire gutters, where have you been all my life?

A cramped utility room with an entire wallcovered with electronic gear, including the DIN rail, which is surrounded bywire gutters Detail of a wire gutter with the cover removed.Numerous large and small wires run along it and exit here and there.

Finally, if you'd like to know what everything on the DIN rail is, from left to right: Ground block, 24v DC disconnect, fridge GFI, spare GFI, USB hub switch, computer switch, +24v block, -24v block, IO daughterboard, 1tb SSD, arm board, modem, 3 USB hubs, 5 relays, AC hot block, AC neutral block, DC-DC power converters, humidity sensor.

Full width of DIN rail.

Monday, March 23rd, 2020
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
3:39 pm
quarantimer: a coronovirus quarantine timer for your things

I am trying to avoid bringing coronovirus into my house on anything, and I also don't want to sterilize a lot of stuff. (Tedious and easy to make a mistake.) Currently it seems that the best approach is to leave stuff to sit undisturbed someplace safe for long enough for the virus to degrade away.

Following that policy, I've quickly ended up with a porch full of stuff in different stages of quarantine, and I am quickly losing track of how long things have been in quarantine. If you have the same problem, here is a solution:

https://quarantimer.app/

Open it on your mobile device, and you can take photos of each thing, select the kind of surfaces it has, and it will track the quarantine time for you. You can share the link to other devices or other people to collaborate.

I anticipate the javascript and css will improve, but it's good enough for now. I will provide this website until the crisis is over. Of course, it's free software and you can also host your own.

If this seems useful, please tell your friends and family about it.

Be well!


This is made possible by my supporters on Patreon, particularly Jake Vosloo.

Friday, February 7th, 2020
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
6:23 pm
arduino-copilot combinators

My framework for programming Arduinos in Haskell has two major improvements this week. It's feeling like I'm laying the keystone on this project. It's all about the combinators now.

Sketch combinators

Consider this arduino-copilot program, that does something unless a pause button is pushed:

paused <- input pin3
pin4 =: foo @: not paused
v <- input a1
pin5 =: bar v @: sometimes && not paused

The pause button has to be checked everywhere, and there's a risk of forgetting to check it, resulting in unexpected behavior. It would be nice to be able to factor that out somehow. Also, notice that it inputs from a1 all the time, but won't use that input when pause is pushed. It would be nice to be able to avoid that unnecessary work.

The new whenB combinator solves all of that:

paused <- input pin3
whenB (not paused) $ do
    pin4 =: foo
    v <- input a1
    pin5 =: bar v @: sometimes

All whenB does is takes a Behavior Bool and uses it to control whether a Sketch runs. It was not easy to implement, given the constraints of Copilot DSL, but it's working. And once I had whenB, I was able to leverage RebindableSyntax to allow if then else expressions to choose between Sketches, as well as between Streams.

Now it's easy to start by writing a Sketch that describes a simple behavior, like turnRight or goForward, and glue those together in a straightforward way to make a more complex Sketch, like a line-following robot:

ll <- leftLineSensed
rl <- rightLineSensed
if ll && rl
    then stop
    else if ll
        then turnLeft
        else if rl
            then turnRight
            else goForward

(Full line following robot example here)

TypedBehavior combinators

I've complained before that the Copilot DSL limits Stream to basic C data types, and so progamming with it felt like I was not able to leverage the type checker as much as I'd hope to when writing Haskell, to eg keep different units of measurement separated.

Well, I found a way around that problem. All it needed was phantom types, and some combinators to lift Copilot DSL expressions.

For example, a Sketch that controls a hot water heater certainly wants to indicate clearly that temperatures are in C not F, and PSI is another important unit. So define some empty types for those units:

data PSI
data Celsius

Using those as the phantom type parameters for TypedBehavior, some important values can be defined:

maxSafePSI :: TypedBehavior PSI Float
maxSafePSI = TypedBehavior (constant 45)

maxWaterTemp :: TypedBehavior Celsius Float
maxWaterTemp = TypedBehavior (constant 35)

And functions like this to convert raw ADC readings into our units:

adcToCelsius :: Behavior Float -> TypedBehavior Celsius Float
adcToCelsius v = TypedBehavior $ v * (constant 200 / constant 1024)

And then we can make functions that take these TypedBehaviors and run Copilot DSL expressions on the Stream contained within them, producing Behaviors suitable for being connected up to pins:

isSafePSI :: TypedBehavior PSI Float -> Behavior Bool
isSafePSI p = liftB2 (<) p maxSafePSI

isSafeTemp :: TypedBehavior Celsius Float -> Behavior Bool
isSafeTemp t = liftB2 (<) t maxSafePSI

(Full water heater example here)

BTW, did you notice the mistake on the last line of code above? No worries; the type checker will, so it will blow up at compile time, and not at runtime.

    • Couldn't match type ‘PSI’ with ‘Celsius’
      Expected type: TypedBehavior Celsius Float
        Actual type: TypedBehavior PSI Float

The liftB2 combinator was all I needed to add to support that. There's also a liftB, and there could be liftB3 etc. (Could it be generalized to a single lift function that supports multiple arities? I don't know yet.) It would be good to have more types than just phantom types; I particularly miss Maybe; but this does go a long way.

So you can have a good amount of type safety while using Copilot to program your Arduino, and you can mix both FRP style and imperative style as you like. Enjoy!


This work was sponsored by Trenton Cronholm and Jake Vosloo on Patreon.

Saturday, February 1st, 2020
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
3:42 pm
arduino-copilot one week along

My framework for programming Arduinos in Haskell in FRP-style is a week old, and it's grown up a lot.

It can do much more than flash a light now. The =: operator can now connect all kinds of FRP Events to all kinds of outputs. There's some type level progamming going on to only allow connections that make sense. For example, arduino-copilot knows what pins of an Adruino support DigitalIO and which support PWM. There are even nice custom type error messages:

demo.hs:7:9: error:
    • This Pin does not support digital IO
    • In a stmt of a 'do' block: a6 =: blinking

I wanted it to be easy to add support to arduino-copilot for using Arduino C libraries from Haskell, and that's proven to be the case. I added serial support last weekend, which is probably one of the harder libraries. It all fell into place once I realized it should not be about individual printfs, but about a single FRP behavior that describes all output to the serial port. This interface was the result:

n <- input a1 :: Sketch (Behavior ADC)
Serial.device =: [Serial.str "a1:", Serial.show n, Serial.char '\n']
Serial.baud 9600

This weekend I've been adding support for the EEPROMex library, and the Functional Reactive Programming approach really shines in stuff like this example, which gathers data from a sensor, logs it to the serial port, and also stores every 3rd value into the EEPROM for later retrival, using the whole EEPROM as a rolling buffer.

v <- input a1 ([10, 20..] :: [ADC])
range <- EEPROM.allocRange sizeOfEEPROM :: Sketch (EEPROM.Range ADC)
range =: EEPROM.sweepRange 0 v @: frequency 3
led =: frequency 3
Serial.device =: [ Serial.show v, Serial.char '\n']
Serial.baud 9600
delay =: MilliSeconds (constant 10000)

There's a fair bit of abstraction in that... Try doing that in 7 lines of C code with that level of readability. (It compiles into 120 lines of C.)

Copilot's ability to interpret the program and show what it would do, without running it on the Adruino, seems more valuable the more complicated the programs become. Here's the interpretation of the program above.

delay:     digitalWrite_13:      eeprom_range_write1:  output_Serial:       
(10000)    (13,false)            --                    (10)                 
(10000)    (13,true)             (0,20)                (20)                 
(10000)    (13,false)            --                    (30)                 
(10000)    (13,false)            --                    (40)                 
(10000)    (13,true)             (1,50)                (50)                 
(10000)    (13,false)            --                    (60)          

Last night I was writing a program that amoung other things, had an event that only happened once every 70 minutes (when the Arduino's micros clock overflows). I didn't have to wait hours staring at the Arduino to test and debug my program, instead I interpreted it with a clock input that overflowed on demand.

(Hmm, I've not actually powered my Arduino on in nearly a week despite writing new Arduino programs every day.)

So arduino-copilot is feeling like it's something that I'll be using soon to write real world Arduino programs. It's certianly is not usable for all Arduino programming, but it will support all the kinds of programs I want to write, and being able to use Functional Reactive Programming will make me want to write them.


Development of arduino-copilot was sponsored by Trenton Cronholm and Jake Vosloo on Patreon.

Saturday, January 25th, 2020
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
3:31 pm
announcing arduino-copilot

arduino-copilot, released today, makes it easy to use Haskell to program an Arduino. It's a FRP style system, and uses the Copilot DSL to generate embedded C code.

gotta blink before you can run

To make your arduino blink its LED, you only need 4 lines of Haskell:

import Copilot.Arduino
main = arduino $ do
    led =: blinking
    delay =: constant (MilliSeconds 100)

Running that Haskell program generates an Arduino sketch in an .ino file, which can be loaded into the Arduino IDE and uploaded to the Arduino the same as any other sketch. It's also easy to use things like Arduino-Makefile to build and upload sketches generated by arduino-copilot.

shoulders of giants

Copilot is quite an impressive embedding of C in Haskell. It was developed for NASA by Galois and is intended for safety-critical applications. So it's neat to be able to repurpose it into hobbyist microcontrollers. (I do hope to get more type safety added to Copilot though, currently it seems rather easy to confuse eg miles with kilometers when using it.)

I'm not the first person to use Copilot to program an Arduino. Anthony Cowley showed how to do it in Abstractions for the Functional Roboticist back in 2013. But he had to write a skeleton of C code around the C generated by Copilot. Amoung other features, arduino-copilot automates generating that C skeleton. So you don't need to remember to enable GPIO pin 13 for output in the setup function; arduino-copilot sees you're using the LED and does that for you.

frp-arduino was a big inspiration too, especially how easy it makes it to generate an Arduino sketch withough writing any C. The "=:" operator in copilot-arduino is copied from it. But ftp-arduino contains its own DSL, which seems less capable than Copilot. And when I looked at using frp-arduino for some real world sensing and control, it didn't seem to be possible to integrate it with existing Arduino libraries written in C. While I've not done that with arduino-copilot yet, I did design it so it should be reasonably easy to integrate it with any Arduino library.

a more interesting example

Let's do something more interesting than flashing a LED. We'll assume pin 12 of an Arduino Uno is connected to a push button. When the button is pressed, the LED should stay lit. Otherwise, flash the LED, starting out flashing it fast, but flashing slower and slower over time, and then back to fast flashing.

{-# LANGUAGE RebindableSyntax #-}
import Copilot.Arduino.Uno

main :: IO ()
main = arduino $ do
        buttonpressed <- readfrom pin12
        led =: buttonpressed || blinking
        delay =: MilliSeconds (longer_and_longer * 2)

This is starting to use features of the Copilot DSL; "buttonpressed || blinking" combines two FRP streams together, and "longer_and_longer * 2" does math on a stream. What a concise and readable implementation of this Arduino's behavior!

Finishing up the demo program is the implementation of longer_and_longer. This part is entirely in the Copilot DSL, and actually I lifted it from some Copilot example code. It gives a reasonable flavor of what it's like to construct streams in Copilot.

longer_and_longer :: Stream Int16
longer_and_longer = counter true $ counter true false `mod` 64 == 0

counter :: Stream Bool -> Stream Bool -> Stream Int16
counter inc reset = cnt
   where
        cnt = if reset then 0 else if inc then z + 1 else z
        z = [0] ++ cnt

This whole example turns into just 63 lines of C code, which compiles to a 1248 byte binary, so there's plenty of room left for larger, more complex programs.

simulating an Arduino

One of Copilot's features is it can interpret code, without needing to run it on the target platform. So the Arduino's behavior can be simulated, without ever generating C code, right at the console!

But first, one line of code needs to be changed, to provide some button states for the simulation:

        buttonpressed <- readfrom' pin12 [False, False, False, True, True]

Now let's see what it does:

# runghc demo.hs -i 5
delay:         digitalWrite_13: 
(2)            (13,false)    
(4)            (13,true)     
(8)            (13,false)    
(16)           (13,true)     
(32)           (13,true)     

Which is exactly what I described it doing! To prove that it always behaves correctly, you could use copilot-theorem.

peek at C

Let's look at the C code that is generated by the first example, of blinking the LED.

This is not the generated code, but a representation of how the C compiler sees it, after constant folding, and some very basic optimisation. This compiles to the same binary as the generated code.

void setup() {
      pinMode(13, OUTPUT);
}
void loop(void) {
      delay(100);
      digitalWrite(13, s0[s0_idx]);
      s0_idx = (++s0_idx) % 2;
}

If you compare this with hand-written C code to do the same thing, this is pretty much optimal!

Looking at the C code generated for the more complex example above, you'll see few unnecessary double computations. That's all I've found to complain about with the generated code. And no matter what you do, Copilot will always generate code that runs in constant space, and constant time.


Development of arduino-copilot was sponsored by Trenton Cronholm and Jake Vosloo on Patreon.

Friday, December 27th, 2019
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
3:55 pm
2020 hindsight

(Someone stumbled upon my 2010 decade retrospective post and suggested I write a followup...)

This has been a big decade for me.

Ten years ago, I'd been in an increasingly stale job for several years too long. I was tired of living in the city, and had a yurt as a weekend relief valve. I had the feeling a big change was coming.

Four months on and I quit my job, despite the ongoing financial crisis making prospects poor for other employment, especially work on free software.

I tried to start a business, Branchable, with liw, based on my earlier ikiwiki project, but it never really took off. However, I'm proud it's still serving the users it did find, 10 years later.

Then, through luck and connections, I found a patch of land in a blank spot in the map with the most absurd rent ever ($5/acre/month). It had a house on it, no running water, barely solar power, a phone line, no cell service or internet, total privacy.

This proved very inspiring. Once again I was hauling water, chopping wood, poking at web pages on the other end of a dialup modem. Just like it was 2000 again. Now I was also hacking by lantern-light until the ancient batteries got so depleted I could hear the voltage regulator crackle with every surge of CPU activity.

I had wanted to learn Haskell, but could never concentrate on it enough. I learned me some Haskell and wrote git-annex, my first real world Haskell program, to help me deal with shuttling data back and forth from civilization on sneakernet.

After two idyllic years of depleting savings, I did a Kickstarter for git-annex and raised not much, but I was now living on very little, so that was a nice windfall. I went full crowdfunding for a couple of years. After a while, I started getting contracting work, supplementing the croudfunding, as git-annex found use in science and education. Both have continued ever since, amazingly.

I was free to do whatever I wanted to. A lot of that was git-annex, with some Debian, and some smaller projects, too many to list here.

Then, mid-decade, I left the Debian project. I'm still sad, still miss everybody, but I also think, had I not been so free, I would not have been able to leave it. It had driven most of my career before this point. I was lucky to be able to leave Debian.

Tuesday, December 10th, 2019
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
3:31 pm
announcing the filepath-bytestring haskell library

filepath-bytestring is a drop-in replacement for the standard haskell filepath library, that operates on RawFilePath rather than FilePath.

The benefit, of course, is speed. "foo" </> "bar" is around 25% faster with the new library. dropTrailingPathSeparator is 120% faster. But the real speed benefits probably come when a program is able to input filepaths as ByteStrings, manipulate them, and operate on the files, all without using String.

It's extensively tested, not only does it run all the same doctests that the filepath library does, but each function is quickchecked to behave the same as the equivilant function from filepath.

While I implemented almost everything, I did leave off some functions that operate on PATH, which seem unlikely to be useful, and the complicated normalise and stuff that uses it.

This work was sponsored by Jake Vosloo on Patron.

Monday, December 9th, 2019
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
4:53 pm
counterpoint on Yahoo Groups archiving

Yahoo Groups being shut down and the data deleted is a big enough story that it's being talked about on the radio. The typical presentation is that they're deleting these mailing list archives, and blocking attempts to save them and so huge amount of things will be lost from the historical record.

That's a common story these days, but it's not entirely accurate in this case. These are mailing lists, so they're not necessarily only archived by Yahoo. Anyone who subscribed to a mailing list may have archived it. I've been on a couple of those mailing lists, and the emails I kept from them are already archived rather well (10+ copies). I probably didn't keep every email, and I probably won't be exhuming those emails to add them to some large archive.org collection of Yahoo Groups. But multiply all the people who subscribed to these lists by all the traffic to them, by the probability that people keep copies of mailing list mails, and there's a huge, well-distributed archive of Yahoo Groups out there.

That ensures some of it will survive in the historical record, probably enough to satisfy a historian.

Probably even after Gmail and the other cloud mail services shut down and delete all their archives.

Previously: I am ArchiveTeam (but not speaking for them above)

Sunday, December 8th, 2019
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
12:04 pm
left handed scissors

They return my hand's grasp, smotheringly close. Was this how it was meant to feel, in a classroom cutting multi-colored construction paper? Not a pain to be gotten through, but comfort, closeness, togetherness. Their design now feels aggressively overdone, broad curve just so around the thumb, as if they might tighten and snap it off. Only too large index finger's knuckle, chafing, provides some relief, some reminder that I shouldn't run.

(Thanks, liw.)

Monday, October 28th, 2019
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
12:21 pm
how I maybe didn't burn out

Last week I found myself in the uncomfortable position of many users strongly disagreeing with a decision I had made about git-annex. It felt much like a pile-on of them vs me, strong language was being deployed, and it was starting to get mentioned in multiple places on the website, in ways I felt would lead to fear, uncertainty, and doubt when other users saw it.

It did not help that I had dental implant surgery recently, and was still more in recovery than I knew when I finally tackled looking at this long thread. So it hit hard.

I've been involved in software projects that have a sometimes adversarial relationship with their users. At times, Debian has been one. I don't know if it is today, but I remember being on #debian and #debian-devel, or debian-user@lists and debian-devel@lists, and seeing two almost entirely diverged communities who were interacting only begrudgingly and with friction.

I don't want that in any of my projects now. My perspective on the history of git-annex is that most of the best developments have come after I made a not great decision or a mistake and got user feedback, and we talked it over and found a way to improve things, leading to a better result than would have been possible without the original mistake, much how a broken bone heals stronger. So this felt wrong, wrong, wrong.

Part of the problem with this discussion was that, though I'd tried to explain the constraints that led to the design decision -- which I'd made well over three years ago -- not everyone was able to follow that or engage with it constructively. Largely, I think because git-annex has a lot more users now, with a wider set of viewpoints. (Which is generally why Debian has to split off user discussions of course.) The users are more fearful of change than earlier adopters tended to be, and have more to lose if git-annex stops meeting their use case. They're invested in it, and so defensive of how they want it to work.

It also doesn't help that, surgery aside, I lack time to keep up with every discussion about git-annex now, if I'm going to also develop it. Just looking at the website tends to eat an entire day with maybe a couple bug fixes and some support answers being the only productive result. So, I have stepped back from following the git-annex website at all, for now. (I'll eventually start looking at parts of it again I'm sure.)

I did find enough value in the thread that I was able to develop a fix that should meet everyone's needs, as I now understand them (released in version 7.20191024). I actually come away with entirely new use cases; I did not realize that some users would perhaps use git-annex for a single large file in a repository otherwise kept entirely in git. Or quite how many users mix storing files in git and git-annex, which I have always seen as somewhat of an exception aside from the odd dotfile.

So the open questions are: How do I keep up with discussion and support traffic now; can I find someone to provide lower-level support and filtering or something? (Good news is, some funding could probably be arranged.) How do I prevent the git-annex community fracturing along users/developer lines as it grows, given that I don't want to work within such a fractured community?

I've been working on git-annex for 9 years this week. Have I avoided burning out? Probably, but maybe too early to tell. I think that being able to ask these questions is a good thing. I'd appreciate hearing from anyone who has grappled with these issues in their own software communities.

Thursday, October 3rd, 2019
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
4:30 pm
Project 62 Valencia Floor Lamp review

From Target, this brass finish floor lamp evokes 60's modernism, updated for the mid-Anthropocene with a touch plate switch.

The integrated microcontroller consumes a mere 2.2 watts while the lamp is turned off, in order to allow you to turn the lamp on with a stylish flick. With a 5 watt LED bulb (sold separately), the lamp will total a mere 7.2 watts while on, making it extremely energy efficient. While off, the lamp consumes a mere 19 kilowatt-hours per year.

clamp multimeter reading 0.02 amps AC, connected to a small circuit board with a yellow capacitor, a coil, and a heat sinked IC visible lamp from rear; a small round rocker switch has been added to the top of its half-globe shade

Though the lamp shade at first appears perhaps flimsy, while you are drilling a hole in it to add a physical switch, you will discover metal, though not brass all the way through. Indeed, this lamp should last for generations, should the planet continue to support human life for that long.

As an additional bonus, the small plastic project box that comes free in this lamp will delight any electrical enthusiast. As will the approximately 1 hour conversion process to delete the touch switch phantom load. The 2 cubic foot of syrofoam packaging is less delightful.

Two allen screws attach the pole to the base; one was missing in my lamp. Also, while the base is very heavily weighted, the lamp still rocks a bit when using the aftermarket switch. So I am forced to give it a mere 4 out of 5 stars.

front view of lit lamp beside a bookcase

Friday, September 27th, 2019
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
3:38 pm
turing complete version numbers

A quick standard for when you want to embed an arbitrary program in the version number of your program.

2   increment the data pointer (to point to the next cell to the right).
3   decrement the data pointer (to point to the next cell to the left).
+   increment (increase by one) the byte at the data pointer.
-   decrement (decrease by one) the byte at the data pointer.
.   output the byte at the data pointer.
4   accept one byte of input, storing its value in the byte at the data pointer.
6   if the byte at the data pointer is zero, then instead of moving the instruction pointer forward to the next command, jump it forward to the command after the matching 9 command.
9   if the byte at the data pointer is nonzero, then instead of moving the instruction pointer forward to the next command, jump it back to the command after the matching 6 command. 

This is simply Brainfuck with operators that are legal in (Debian) version numbers kept as-is, and some numbers replacing the rest.

Note that all other operators are ignored as usual. In particular, 1 and 8 are ignored, which make it easy to build version number programs that compare properly with past versions. And in some cases, adding 1 or 8 will be needed to make a particular program be a properly formatted version number.

For example, an infinite loop version number is:

1+69

A nice short hello world is:

1+6-6336+6-8-1-29-6333999222-92-1.1-1-1-8.2.8.2.3333-1.3+1.22222.2.33.3-1.1

Licensing: Yes, there should also be a way to embed a license in a version ... Oh, I mean to say, the Wikipedia excerpt above is CC-BY-SA, and the hello world is based on https://esolangs.org/wiki/Hello_world_program_in_esoteric_languages

Previously: a brainfuck monad

Saturday, September 21st, 2019
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
6:29 pm
how to detect chef

If you want your program to detect when it's being run by chef, here's one way to do that.

sleep 1 while $ENV{PATH} =~ m#chef[^:]+/bin#;

This works because Chef's shell_out adds Gem.bindir to PATH, which is something like /opt/chefdk/embedded/bin.

You may want to delete the "sleep", which will make it run faster.

Would I or anyone ever really do this? Chef Inc's management seems determined to test the question, don't they.

Wednesday, August 21st, 2019
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
12:28 pm
releasing two haskell libraries in one day: libmodbus and git-lfs

The first library is a libmodbus binding in haskell.

There are a couple of other haskell modbus libraries, but none that support serial communication out of the box. I've been using a python library to talk to my solar charge controller, but it is not great at dealing with the slightly flakey interface. The libmodbus C library has features that make it more robust, and it also supports fast batched reads.

So a haskell interface to it seemed worth starting while I was doing laundry, and then for some reason it seemed worth writing a whole bunch more FFIs that I may never use, so it covers libmodbus fairly extensively. 660 lines of code all told.

Writing a good binding to a C library has art to it. I've seen ones that are so close you feel you're writing C and not haskell. On the other hand, some are so far removed from the underlying library that its documentation does not carry over at all.

I tried to strike a balance. Same function names so the extensive libmodbus documentation is easy to refer to while using it, but plenty of haskell data types so you won't mix up the parity with the stop bits.

And while it uses a mutable vector under the hood as the buffer for the FFI interface, so it can be just as fast as the C library, I also made functions for reading stuff like registers and coils be polymorphic so easier data types can be used at the expense of a bit of extra allocation.

The big win in this haskell binding is that you can leverage all the nice haskell libraries for dealing with binary data to parse the modbus data, rather than the ad-hoc integer and float conversion stuff from the C library.

For example, the Epever solar charge controller has its own slightly nonstandard way to represent 16 bit and 32 bit floats. Using the binary library to parse its registers in applicative style came out quite nice:

data Epever = Epever
    { pv_array_voltage :: Float
    , pv_array_current :: Float
    , pv_array_power :: Float
    , battery_voltage :: Float
    } deriving (Show)

getEpever :: Get Epever
getEpever = Epever
    <$> epeverfloat  -- register 0x3100
    <*> epeverfloat  -- register 0x3101
    <*> epeverfloat2 -- register 0x3102 (low) and 0x3103 (high)
    <*> epeverfloat  -- register 0x3104
 where
    epeverfloat = decimals 2 <$> getWord16host
    epeverfloat2 = do
        l <- getWord16host
        h <- getWord16host
        return (decimals 2 (l + h*2^16))
    decimals n v = fromIntegral v / (10^n)

The second library is a git-lfs implementation in pure Haskell.

Emphasis on the pure -- there is not a scrap of IO code in this library, just 400+ lines of data types, parsing, and serialization.

I wrote it a couple weeks ago so git-annex can store files in a git-lfs remote. I've also used it as a git-lfs server, mostly while exploring interesting edge cases of git-lfs.


This work was sponsored by Jake Vosloo on Patreon.

Tuesday, July 2nd, 2019
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
4:53 pm
custom type checker errors for propellor

Since propellor is configured by writing Haskell, type errors are an important part of its interface. As more type level machinery has been added to propellor, it's become more common for type errors to refer to hard to understand constraints. And sometimes simple mistakes in a propellor config result in the type checker getting confused and spewing an error that is thousands of lines of gobbledygook.

Yesterday's release of the new type-errors library got me excited to improve propellor's type errors.

Most of the early wins came from using ghc's TypeError class, not the new library. I wanted custom type errors that were able to talk about problems with Property targets, like these:

    • ensureProperty inner Property is missing support for: 
    FreeBSD

    • This use of tightenTargets would widen, not narrow, adding: 
        ArchLinux + FreeBSD

    • Cannot combine properties:
        Property FreeBSD
        Property HasInfo + Debian + Buntish + ArchLinux

So I wrote a type-level pretty-printer for propellor's MetaType lists. One interesting thing about it is that it rewrites types such as Targeting OSDebian back to the Debian type alias that the user expects to see.

To generate the first error message above, I used the pretty-printer like this:

(TypeError
    ('Text "ensureProperty inner Property is missing support for: "
        ':$$: PrettyPrintMetaTypes (Difference (Targets outer) (Targets inner))
    )
)

Often a property constructor in propellor gets a new argument added to it. A propellor config that has not been updated to include the new argument used to result in this kind of enormous and useless error message:

    • Couldn't match type ‘Propellor.Types.MetaTypes.CheckCombinable
                             (Propellor.Types.MetaTypes.Concat
                                (Propellor.Types.MetaTypes.NonTargets y0)
                                (Data.Type.Bool.If
                                   (Propellor.Types.MetaTypes.Elem
                                      ('Propellor.Types.MetaTypes.Targeting 'OSDebian)
                                      (Propellor.Types.MetaTypes.Targets y0))
                                   ('Propellor.Types.MetaTypes.Targeting 'OSDebian
                                      : Data.Type.Bool.If
                                          (Propellor.Types.MetaTypes.Elem
                                             ('Propellor.Types.MetaTypes.Targeting 'OSBuntish)
    -- many, many lines elided
    • In the first argument of ‘(&)’, namely
        ‘props & osDebian Unstable’

The type-errors library was a big help. It's able to detect when the type checker gets "stuck" reducing a type function, and is going to dump it all out to the user. And you can replace that with a custom type error, like this one:

    • Cannot combine properties:
        Property <unknown>
        Property HasInfo + Debian + Buntish + ArchLinux + FreeBSD
        (Property <unknown> is often caused by applying a Property constructor to the wrong number of arguments.)
    • In the first argument of ‘(&)’, namely
        ‘props & osDebian Unstable’

Detecting when the type checker is "stuck" also let me add some custom type errors to handle cases where type inference has failed:

    • ensureProperty outer Property type is not able to be inferred here.
      Consider adding a type annotation.
    • When checking the inferred type
        writeConfig :: forall (outer :: [Propellor.Types.MetaTypes.MetaType]) t.

    • Unable to infer desired Property type in this use of tightenTargets.
      Consider adding a type annotation.

Unfortunately, the use of TypeError caused one problem. When too many arguments are passed to a property constructor that's being combined with other properties, ghc used to give its usual error message about too many arguments, but now it gives the custom "Cannot combine properties" type error, which is not as useful.

Seems likely that's a ghc bug but I need a better test case to make progress on that front. Anyway, I decided I can live with this problem for now, to get all the other nice custom type errors.

The only other known problem with propellor's type errors is that, when there is a long list of properties being combined together, a single problem can result in a cascade of many errors. Sometimes that also causes ghc to use a lot of memory. While custom error messages don't help with this, at least the error cascade is nicer and individual messages are not as long.

Propellor 5.9.0 has all the custom type error messages discussed here. If you see a hard to understand error message when using it, get in touch and let's see if we can make it better.


This was sponsored by Jake Vosloo and Trenton Cronholm on Patreon.

Saturday, June 15th, 2019
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
12:02 pm
hacking water

From water insecurity to offgrid, solar pumped, gravity flow 1000 gallons of running water.

I enjoy hauling water by hand, which is why doing it for 8 years was not really a problem. But water insecurity is; the spring has been drying up for longer periods in the fall, and the cisterns have barely been large enough to get through.

And if I'm going to add storage, it ought to be above the house, so it can gravity flow. And I have this 100 watt array of 20 year old solar panels sitting unused after my solar upgrade. And a couple of pumps for a pressure tank system that was not working when I moved in. And I stumbled across an odd little flat spot halfway up the hillside. And there's an exposed copper pipe next to the house's retaining wall; email to Africa establishes that it goes down and through the wall and connects into the plumbing.

a solarpanel with a large impact crater; the glass has cracked into thousands ofpeices but is still hanging together barely perceptable flat spot on treecovered hillside a copper pipe sheathed inblack plastic curves out of the ground next to a wall

So I have an old system that doesn't do what I want. Let's hack the system..

(This took a year to research and put together, including learning a lot about plumbing.)

Run a cable from the old solar panels 75 feet over to the spring. Repurpose an old cooler as a pumphouse, to keep the rain off the Shurflow pump, and with the opening facing so it directs noise away from living areas. Add a Shurflow 902-200 linear current booster to control the pump.

red cooler attached to a tree with apump in it. water is streaming out of one of the two pipes attached to it circuit board with terminalslabeled PUMP, PV, HIGH, LOW, GND, FLOAT SWITCH 50 gallon water barrelperched on a hillside with some hoses connected to it

Run a temporary pipe up to the logging road, and verify that the pump can just manage to push the water up there.

Sidetrack into a week spent cleaning out and re-sealing the spring's settling tank. This was yak shaving, but it was going to fail. Build a custom ladder because regular ladders are too wide to fit into it. Flashback to my tightest squeezes from caving. Yuurgh.

very narrow concretewater tank with its concrete lid opened and a rough wooden ladder stickingout of it interior of water tankdrained with muck covering the bottom and plaster flaking from walls interior of watertank with walls bright and new, water level sensors and pipe to pump

Install water level sensors in the settling tank, cut a hole for pipe, connect to pumphouse.

Now how to bury 250 feet of PEX pipe a foot deep up a steep hillside covered in rock piles and trees that you don't want to cut down to make way for equipment? Research every possibility, and pick the one that involves a repurposed linemans's tool resembling a medieval axe.

hillside strewn in large rockswith trees wherever there are not rocks just unboxedtrenching tool looks like a large black metal spatula 30 feet of very narrow trenchcomes out of the woods and along the side of the house past the satelliteinternet dish lines drawnover a photo of the hillside show the pipe's curving route to the top

Dig 100 feet of 1 inch wide trench in a single afternoon by hand. Zeno in on the rest of the 300 foot run. Gain ability to bury underground cables without raising a sweat as an accidental superpower. Arms ache for a full month afterwards.

Connect it all up with a temporary water barrel, and it works! Gravity flow yields 30 PSI!

Pressure-test the copper pipe going into the house to make sure it's not leaking behind the retaining wall. Fix all the old leaky plumbing and fixtures in the house.

water pressure guage connected toPEX pipe coming out of trench and connecting to copper pipe that goes intohouse modern restaurant-style sprung archedfaucet with water flowing into the kitchen sink

Clear a 6 foot wide path through the woods up the hill and roll up two 550 gallon Norwesco water tanks. Haul 650 pounds of sand up the hill, by hand, one 5 gallon bucket at a time. Level and prepare two 6 foot diameter pads.

Joey standing in front of a black 4x4pickup truck with a large white 550 gallon water tank on its side in thebed and arching high above Joey from the back as he rolls watertank up a forested hill six foot circle marked off with ropeand filled with sand; a water tank is in the background and tools arestrewn around the cramped worksite

Build a buried manifold with valves turned by water meter key. Include a fire hose outlet just in case.

in a hole inthe ground between two water tanks is a complex assembly of blue pipes andbrass fittings, with several valves close-up of old cracked solar panel the two tanks installed, high on a hillside

Begin filling the tanks, unsure how long it will take as the pump balances available sunlight and spring flow.

Saturday, June 8th, 2019
LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.
10:49 am
the most important missing unicode extension

Unicode allows you to write any symbol in use in the world, and if one is somehow missing from Unicode, it'll get added in a future version of the specification. Right?

Here's a symbol we all know well: ￿
Aka, "box with 4 small F's in it".

But that's not the only such symbol, there was a whole class of them in common use in the early 2000's, so common that they were called "tofu" when grouped together.

Modern unicode systems cannot display those historical characters!

Clearly this needs to be fixed with an extension to the Unicode standard. I propose TOFU, a combining character that makes whatever it's combined with be displayed as a font fallback box glyph containing its Unicode code point.

[ << Previous 20 ]

LJ.Rossia.org makes no claim to the content supplied through this journal account. Articles are retrieved via a public feed supplied by the site for this purpose.