799 lines
33 KiB
Org Mode
799 lines
33 KiB
Org Mode
* [2024-06-28 Fri]
|
|
** well, i'm setting up sbcl and sly?
|
|
because i imagine using Common Lisp for my attempt at building screen
|
|
sharing for the XR glasses
|
|
|
|
for the cool runtime access, right?
|
|
** well, just flake shell and (use-package sly) is enough
|
|
for .lisp files to get stuff
|
|
|
|
yes, i'm still using C-M-f and stuff to move around sexps but things are fine
|
|
|
|
so, what's next?
|
|
well, this is nice high level description
|
|
dbus
|
|
https://www.freedesktop.org/wiki/Software/dbus/
|
|
|
|
and here's something to start usage
|
|
https://blog.macrolet.net/posts/DBus-and-PolicyKit-from-Common-Lisp.html
|
|
from https://github.com/death/dbus
|
|
|
|
and even lots of examples, but i think i will need a big book on Common Lisp to learn new things?
|
|
* [2024-06-29 Sat]
|
|
** going through the dbus guide, sounds neat
|
|
https://develop.kde.org/docs/features/d-bus/introduction_to_dbus/
|
|
|
|
that qdbus and qdbus-viewer are in
|
|
#+begin_src bash
|
|
$ nix shell nixpkgs#kdePackages.qttools
|
|
#+end_src
|
|
|
|
** now let's read about calling methods
|
|
|
|
* the receiving of the signals
|
|
|
|
* all the different tabs i had
|
|
** overall freedesktop portal dbus api docs
|
|
https://docs.flatpak.org/en/latest/portal-api-reference.html#gdbus-org.freedesktop.portal.ScreenCast
|
|
|
|
** searching github for 'get-managed-objects usage
|
|
https://github.com/search?q=get-managed-objects&type=code
|
|
|
|
** searchin github for 'define-singlan-handler usage examples
|
|
https://github.com/search?q=dbus%3Adefine-dbus-signal-handler&type=code
|
|
|
|
** example in cl-notify of calling a method 'with-introspected-object
|
|
https://github.com/Lautaro-Garcia/cl-notify/blob/main/src/dbus-protocol.lisp#L27
|
|
#+begin_src lisp
|
|
(defmacro with-dbus-method-inovaction ((result-var method &rest args) &body body)
|
|
(alexandria:with-gensyms (bus notifications-object)
|
|
`(dbus:with-open-bus (,bus (dbus:session-server-addresses))
|
|
(dbus:with-introspected-object (,notifications-object ,bus "/org/freedesktop/Notifications" "org.freedesktop.Notifications")
|
|
(let ((,result-var (,notifications-object "org.freedesktop.Notifications" ,method ,@args)))
|
|
,@body)))))
|
|
#+end_src
|
|
|
|
** example in cl-notify of working with signals
|
|
https://github.com/Lautaro-Garcia/cl-notify/blob/main/src/signals.lisp
|
|
|
|
they register single handler, and add callbacks dynamically & call
|
|
them from the registered function
|
|
|
|
** dbus library example for publishing objects
|
|
https://github.com/death/dbus/blob/master/examples/publish.lisp
|
|
|
|
** my request for help with interactive async calls
|
|
https://github.com/death/dbus/issues/31
|
|
|
|
** old ticket on dbus repo about enabling introspection
|
|
https://github.com/death/dbus/issues/23
|
|
|
|
** example of signal-handler in stumpwm-thingy
|
|
https://github.com/lokedhs/stumpwm-dbus/blob/9cf0b52876111e777da27a42e3c81269b97c0005/src/pidgin.lisp#L16
|
|
|
|
** ref for symbold in dbus package
|
|
https://quickref.common-lisp.net/dbus.html#The-dbus_002fconnections-system
|
|
|
|
** blog article with basics of usage
|
|
https://blog.macrolet.net/posts/DBus-and-PolicyKit-from-Common-Lisp.html
|
|
|
|
** big article about iolib
|
|
https://pages.cs.wisc.edu/~psilord/blog/data/iolib-tutorial/tutorial.html
|
|
|
|
it's the one that's used to create multiplexer for communication
|
|
i tried to figure out what it means to have event-loops
|
|
|
|
** docs on dbus Screenshots portal
|
|
https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.Screenshot.html
|
|
|
|
the methods and stuff
|
|
|
|
** docs on dbus type system
|
|
https://dbus.freedesktop.org/doc/dbus-specification.html#type-system
|
|
|
|
i.e how to understand types of methods and properties
|
|
|
|
#+begin_src lisp
|
|
CL-USER> (dbus:sigexp "xsaysas")
|
|
(:INT64 :STRING (:ARRAY :BYTE) :STRING (:ARRAY :STRING))
|
|
#+end_src
|
|
|
|
function signature explanation can show what should lisp types be
|
|
https://github.com/death/dbus/issues/17#issuecomment-418920440
|
|
|
|
** example of flameshot how they get Screenshot via dbus
|
|
https://github.com/flameshot-org/flameshot/blob/c1dac52231024174faa68a29577129ebca125dff/src/utils/screengrabber.cpp#L59
|
|
|
|
** dbus methods for Notification
|
|
https://specifications.freedesktop.org/notification-spec/notification-spec-latest.html#signals
|
|
|
|
|
|
** dbus methods of ScreenCast
|
|
https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.ScreenCast.html
|
|
how to start them etc
|
|
|
|
** dbus spec on Request & Response for the async methods
|
|
https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.Request.html#org-freedesktop-portal-request
|
|
|
|
** repo for xdg-portal for wayland
|
|
https://github.com/emersion/xdg-desktop-portal-wlr
|
|
|
|
** the python snippet to check that ScreenCast via xdg-portal works
|
|
https://gitlab.gnome.org/-/snippets/19
|
|
|
|
* [2024-07-20 Sat]
|
|
|
|
** and now i want to try to get display of stuff into gtk window, i guess
|
|
|
|
** doc about cl bindings to gtk3
|
|
https://www.crategus.com/books/cl-gtk/gtk-tutorial.html#example-window-simple-c
|
|
|
|
** well, i guess i could go and try to read guide on pipewire with C examples
|
|
https://docs.pipewire.org/page_tutorial.html
|
|
this is tutorial for the system
|
|
|
|
and here is example program that maybe does similar thing
|
|
https://docs.pipewire.org/video-play_8c-example.html
|
|
|
|
* [2024-07-28 Sun]
|
|
|
|
** ok, now i'm trying to get frames from the pipewire 'out' node
|
|
and one option i'm exploring is gstreamer
|
|
|
|
which is a framework for creating applications that stream video / audio
|
|
with some kind of ability to define \ declare pipeline and then start
|
|
and then use api somehow
|
|
|
|
there's c api, and gobject accessibility layer
|
|
so either c to common lisp, or gobject common lisp thingy maybe exist
|
|
|
|
** reading docs
|
|
https://gstreamer.freedesktop.org/documentation/frequently-asked-questions/using.html?gi-language=c
|
|
|
|
added via
|
|
gst_all_1.gstreamer
|
|
|
|
** more readin
|
|
gst-launch-1.0 videotestsrc ! videoconvert ! autovideosink
|
|
this is test for how it's installed
|
|
|
|
needed to add plugins
|
|
|
|
https://nixos.wiki/wiki/GStreamer
|
|
|
|
** i guess i can try to go with the tutorials?
|
|
https://gstreamer.freedesktop.org/documentation/tutorials/index.html?gi-language=c
|
|
|
|
but with common lisp over gobject?
|
|
i think i'm already using this in the gtk bindings playground? maybe not
|
|
https://cl-gtk2.common-lisp.dev/doc/gobject/Creating-GObjects-classes-and-implementing-GInterfaces.html
|
|
#+begin_quote
|
|
GObject binding at the moment provides only limited scenarios of creating GObject classes.
|
|
#+end_quote
|
|
|
|
** and there's something
|
|
https://github.com/BohongHuang/cl-gobject-introspection-wrapper
|
|
|
|
which is based on older
|
|
|
|
https://github.com/andy128k/cl-gobject-introspection
|
|
|
|
https://search.nixos.org/packages?channel=unstable&from=0&size=50&sort=relevance&type=packages&query=gobject
|
|
|
|
|
|
they are packaged in nix, this is just unbelievable how much is done by humanity, omg
|
|
|
|
* [2024-08-15 Thu]
|
|
|
|
** getting back after a long while.
|
|
so, i was at the creating a window, not knowing how to stream video from pipewire out node
|
|
so. what to do, huh
|
|
i think gtk3 bindings library was separate from gobject and glib
|
|
|
|
but maybe it doesn't matter?
|
|
or maybe i should figure out glib bindings and use gtk3 though it?
|
|
|
|
** well, there's something simple with opening a window
|
|
[[file:gtk-playground.lisp::defun example-window-simple (]]
|
|
|
|
** and something extra simple with gstreamer opening separate window for test video source
|
|
[[file:gstreamer/tutorial-hello-world.lisp]]
|
|
|
|
** maybe try to open gstreamer window with screen share?
|
|
|
|
ok, when i try example of pipewire window on screenshare source
|
|
i get error that pipewiresrc is not found
|
|
and that's right, i used system-wide gst-launch-1.0
|
|
and when i tried with one in flake shell - it also didn't work
|
|
|
|
so. tried removing all from shell env, then not found namespace Gst
|
|
(defvar *gstreamer* (gir:require-namespace "Gst"))
|
|
|
|
so. yeah. maybe i need to install dev headers or something?
|
|
|
|
** so. for moving the video around. chatgpt recommended something named 'videobox'
|
|
|
|
like i can declare it in the pipeline and then get reference to it and use to set video coordinates, huh?
|
|
https://gstreamer.freedesktop.org/documentation/videobox/index.html?gi-language=python
|
|
the thing exists but i have zero idea whether it's accessible in a way that's recommended
|
|
#+begin_src lisp
|
|
(defun for-screencast ()
|
|
(gir:invoke (*gstreamer* 'init) '())
|
|
(let* ((pipeline (gir:invoke (*gstreamer* 'parse-launch)
|
|
"pipewiresrc path=49 ! videoconvert ! videobox top=0 left=0 right=0 bottom=0 ! videoconvert ! autovideosink"
|
|
))
|
|
(videobox (gir:invoke (pipeline 'get-element-by-name) "videobox")))
|
|
;; Set the initial state to playing
|
|
(gir:invoke (pipeline 'set-state) (gir:nget *gstreamer* "State" :playing))
|
|
|
|
;; Function to move the video
|
|
(defun move-video (top left)
|
|
(gir:invoke (videobox 'set-property) 'top top)
|
|
(gir:invoke (videobox 'set-property) 'left left))
|
|
|
|
;; Example: Move the video 10 pixels down and 20 pixels to the right
|
|
(move-video 10 20)))
|
|
#+end_src
|
|
|
|
and similar advice from claude, huh
|
|
#+begin_src lisp
|
|
(defun for-screencast ()
|
|
(gir:invoke (*gstreamer* 'init) '())
|
|
(let* ((pipeline (gir:invoke (*gstreamer* 'parse-launch)
|
|
"pipewiresrc path=49 ! videoconvert ! videobox name=move ! videoscale ! autovideosink"))
|
|
(move-element (gir:invoke (pipeline 'get-by-name) "move")))
|
|
(gir:invoke (pipeline 'set-state) (gir:nget *gstreamer* "State" :playing))
|
|
(values pipeline move-element)))
|
|
|
|
(defun move-video (move-element left top)
|
|
(gir:set-property move-element "left" left)
|
|
(gir:set-property move-element "top" top))
|
|
|
|
;; Usage:
|
|
(multiple-value-bind (pipeline move-element) (for-screencast)
|
|
;; Move video 50 pixels to the right and 30 pixels down
|
|
(move-video move-element 50 30)
|
|
;; Your main loop or other logic here
|
|
)
|
|
#+end_src
|
|
|
|
let's try this out
|
|
|
|
* [2024-08-16 Fri]
|
|
|
|
** new day of the vacation
|
|
the experiment with moving the video, right?
|
|
|
|
could it be that both examples are fake? i don't seem to find both 'get-element things
|
|
|
|
trying to find anything about dynamism in gstreamer
|
|
https://coaxion.net/blog/2014/01/gstreamer-dynamic-pipelines/
|
|
looks like this is about changing elements of the pipeline
|
|
|
|
and it's possible that parameters of the pipeline elements (pads?) are also static
|
|
|
|
** options for transforming frames?
|
|
*** videotransform : A combination of scaling, 90-degree-step rotation,
|
|
https://github.com/Freescale/gstreamer-imx/blob/master/README.md
|
|
horizontal/vertical flipping, and color space conversion
|
|
operations. These elements are able to render video overlay
|
|
compositions from GstVideoOverlayCompositionMeta data. They also
|
|
implement the GstVideoDirection interface and have a "video-direction"
|
|
property that handles rotation and flipping. Through this property, it
|
|
is possible to configure these elements to auto-rotate images
|
|
according to the information in image-orientation tags.
|
|
*** glimagesink
|
|
https://gstreamer.freedesktop.org/documentation/opengl/glimagesink.html?gi-language=python
|
|
gst-launch-1.0 -v videotestsrc ! video/x-raw ! glimagesink
|
|
has error
|
|
ERROR: from element /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink: Failed to create EGLDisplay from native display
|
|
Additional debug info:
|
|
../ext/gl/gstglimagesink.c(1136): _ensure_gl_setup (): /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink
|
|
ERROR: pipeline doesn't want to preroll.
|
|
|
|
so maybe opengl is not available? how to check?
|
|
*** gltransformation
|
|
https://gstreamer.freedesktop.org/documentation/opengl/gltransformation.html?gi-language=python
|
|
|
|
here examples seem to be for images, not video
|
|
and same error, let's figure out the opengl check
|
|
** somewhat categorized things?
|
|
* [2024-08-21 Wed]
|
|
** visiting sister, returning back
|
|
last time i got stuck on video displaying frozen when i tried to add appsink
|
|
|
|
just re-checking my basic gstreamer pipeline and screencast start - thing works
|
|
but i want to clean it up i guess, to allow for simpler retries?
|
|
** so, for the appsnk there are signals
|
|
https://gstreamer.freedesktop.org/documentation/app/appsink.html?gi-language=python#appsink::new-sample
|
|
|
|
new-sample
|
|
|
|
#+begin_src python
|
|
def new_sample_callback (appsink, udata):
|
|
#python callback for the 'new-sample' signal
|
|
#+end_src
|
|
|
|
Signal that a new sample is available.
|
|
|
|
but how do i connect to a callback from common lisp and gobject introspection
|
|
https://github.com/andy128k/cl-gobject-introspection
|
|
does the gitlab repo has info on callbacks?
|
|
|
|
can't seem to call method 'pull-sample'
|
|
maybe article will help?
|
|
https://gstreamer.freedesktop.org/documentation/application-development/advanced/pipeline-manipulation.html?gi-language=python#grabbing-data-with-appsink
|
|
|
|
well, this looks like method on gst namespace? "gst_app_sink_pull_sample()"
|
|
https://gstreamer.freedesktop.org/documentation/applib/gstappsink.html?gi-language=python#gst_app_sink_pull_sample
|
|
|
|
there's article for python bindings
|
|
https://lifestyletransfer.com/how-to-use-gstreamer-appsink-in-python/
|
|
* [2024-08-22 Thu]
|
|
** hey, i think this is progress
|
|
i am able to call appsink 'pull-sample after all (for some reason, i don't know what i was doing differently)
|
|
|
|
new error:
|
|
>>> got to process sample with #<STRUCT-INSTANCE {10128D02F3}>
|
|
but in the code for processing of the sample some type errors
|
|
|
|
Backtrace:
|
|
0: ((:METHOD GIR::THIS-OF (T)) T) [fast-method]
|
|
1: (GIR:PROPERTY T DATA)
|
|
2: (PROCESS-FRAME #<GIR::STRUCT-INSTANCE {100BCDAB73}>)
|
|
3: ((LAMBDA (APPSINK) :IN START-PROCESSING) #<GIR::OBJECT-INSTANCE {100BBE02D3}>)
|
|
|
|
ok, but when i have struct instance, how do i introspect it?
|
|
** let's just go and read documentation for the type that's returned from pull-sample, sure
|
|
also there's that python code, right?
|
|
doc 1: get sample on appsink https://people.freedesktop.org/~tsaunier/documentation/gst-plugins-base-app-1.0/gstappsink.html?gi-language=c#gst_app_sink_pull_sample
|
|
doc 2: GstSample class https://people.freedesktop.org/~tsaunier/documentation/gstreamer-1.0/gstsample.html?gi-language=c#GstSample
|
|
|
|
so on GstSample everything is taken as results of funciton calls? let's try that
|
|
|
|
well, maybe i don't need get-info on sample, it returns null, maybe i need it on buffer?
|
|
doc 3: get-buffer on sample https://people.freedesktop.org/~tsaunier/documentation/gstreamer-1.0/gstsample.html?gi-language=python#gst_sample_get_buffer
|
|
doc 4: buffer class info https://people.freedesktop.org/~tsaunier/documentation/gstreamer-1.0/gstbuffer.html?gi-language=python#gst_buffer_peek_memory
|
|
|
|
** i'm recommended to look into GstBuffer.map
|
|
https://people.freedesktop.org/~tsaunier/documentation/gstreamer-1.0/gstbuffer.html?gi-language=python#gst_buffer_map
|
|
|
|
it returns some https://people.freedesktop.org/~tsaunier/documentation/gstreamer-1.0/gstmemory.html?gi-language=python#GstMapInfo
|
|
|
|
** struggling, searching for answers
|
|
https://github.com/andy128k/cl-gobject-introspection/issues/73#issuecomment-755316735
|
|
Examples of inspection of a repository can be found in test files.
|
|
|
|
https://github.com/andy128k/cl-gobject-introspection/blob/master/test/test1.lisp
|
|
|
|
and i mean it's possible that referencing members of a structure is not implemented?
|
|
|
|
so, huh
|
|
using internal functions
|
|
#+begin_src lisp
|
|
(data (gir:fields-dict-of (gir:struct-class-of map-info)))
|
|
#+end_src
|
|
i see
|
|
>>>> and data has size NIL and buf ((memory . #<FIELD-INFO {100FEB4083}>)
|
|
(flags . #<FIELD-INFO {100FEB4153}>)
|
|
(data . #<FIELD-INFO {100FEB4223}>)
|
|
(size . #<FIELD-INFO {100FEB42F3}>)
|
|
(maxsize . #<FIELD-INFO {100FEB43C3}>)
|
|
(user_data . #<FIELD-INFO {100FEB4493}>)
|
|
(_gst_reserved
|
|
. #<FIELD-INFO {100FEB4563}>))
|
|
; Evaluation aborted on T
|
|
CAPTURING-FRAMES>
|
|
|
|
so. there are data and size fields.
|
|
|
|
maybe i can try with
|
|
|
|
** well, for some reason data is not avaiable?
|
|
(let* ((buffer (gir:invoke (sample 'get-buffer)))
|
|
(info (gir:invoke (sample 'get-info)))
|
|
(caps (gir:invoke (sample 'get-caps)))
|
|
(caps-struct (gir:invoke (caps 'get-structure) 0))
|
|
(width (gir:invoke ((gir:invoke (caps-struct 'get-value) "width") 'get-int)))
|
|
(height (gir:invoke ((gir:invoke (caps-struct 'get-value) "height") 'get-int)))
|
|
(buf-size (gir:invoke (buffer 'get-size)))
|
|
(data-copy (gir:invoke (buffer 'extract-dup) 0 buf-size)))
|
|
(format t ">>>> at buffer got size ~A and tried to copy ~A~%" buf-size data-copy)
|
|
;; ok, we have buffer, let's try to get copy
|
|
|
|
;; (multiple-value-bind (found map-info) (gir:invoke (buffer 'map) (gir:nget *gstreamer* "MapFlags" :read))
|
|
;; (format t ">>>> the buffer has offset ~A~%" (gir:field buffer 'offset))
|
|
;; (format t ">>>> what's in struct? ~A~%" (gir:invoke (caps-struct 'to-string)))
|
|
;; (format t ">>>> the fields are width ~A, height ~A~%" width height)
|
|
;; (format t ">>>> and map result are ~A and ~A~%" found map-info)
|
|
;; (when found
|
|
;; (let ((data (gir:field map-info 'memory))
|
|
;; (size (gir:field map-info 'size)))
|
|
;; (format t ">>>> and struct info? ~A~%" map-info)
|
|
;; (gir:invoke (buffer 'unmap) map-info)
|
|
;; (format t ">>>> and data has size ~A and memory is ~A~%" size (or data "empty data")))))
|
|
)
|
|
|
|
data-copy is NIL
|
|
yep. mda.
|
|
|
|
** well, trying to copy as in python example didn't work either
|
|
(buf-size (gir:invoke (buffer 'get-size)))
|
|
(data-copy (gir:invoke (buffer 'extract-dup) 0 buf-size))
|
|
|
|
https://github.com/jackersson/gst-python-tutorials/blob/master/launch_pipeline/run_appsink.py#L49
|
|
|
|
yeah, it's possible that gir library doesn't work well? even if on c level all is ok?
|
|
|
|
* [2024-08-23 Fri]
|
|
|
|
** well, problem is in trying to read gir:map-c-array
|
|
it gets length nil and fails
|
|
|
|
GIR::MAP-C-ARRAY
|
|
|
|
so what i'm using:
|
|
https://github.com/andy128k/cl-gobject-introspection
|
|
and there's wrapper around it
|
|
https://github.com/bohonghuang/cl-gobject-introspection-wrapper
|
|
|
|
but there is also
|
|
https://github.com/kat-co/gir2cl
|
|
maybe it will work?
|
|
or maybe i'll figure out what's up
|
|
|
|
** well, trying to check available slots on that array type
|
|
(mapcar #'c2mop:slot-definition-name (c2mop:class-slots (class-of c-array-type)))
|
|
=> (PARAM-TYPE FIXED-SIZE ZERO-TERMINATED? LENGTH)
|
|
|
|
and how do i get their values? if i don't know name of the :reader ?
|
|
( from article on https://riptutorial.com/common-lisp/example/9826/obtain-the-slot-names-of-a-class )
|
|
|
|
but yeah. huh.
|
|
|
|
** let's write up the problem?
|
|
I'm trying to use gstreamer - a framework for video\audio pipelines with gobject api
|
|
|
|
the hope is to use their 'appsink' to get video frame data into the program i'm writing
|
|
https://people.freedesktop.org/~tsaunier/documentation/gstreamer-1.0/gstbuffer.html?gi-language=python#gst_buffer_map
|
|
|
|
i'll link the code in case it will be needed for replication,
|
|
right now it seems that by doing gir calls i've successfully set up pipeline
|
|
and receiving callbacks with the buffers
|
|
|
|
the problem i get is when i try to access 'data
|
|
and it seems to come from GIR:MAP-C-ARRAY
|
|
where length of the parameter c-array-type is NIL
|
|
|
|
in #'map-c-array it comes by calling
|
|
#+begin_src lisp
|
|
(defun c-array-length (c-array-type)
|
|
(with-slots (param-type fixed-size zero-terminated? length)
|
|
c-array-type
|
|
(if fixed-size
|
|
(if (and length (< length fixed-size))
|
|
length
|
|
fixed-size)
|
|
(if length
|
|
(+ length (if zero-terminated? 1 0))))))
|
|
#+end_src
|
|
|
|
and manually checking slots of the 'c-array-type
|
|
it has NIL in length
|
|
|
|
right now I don't know whether it's a problem on the side of cl-gobject-introspection
|
|
in how it's initializing the c-array-type
|
|
or maybe it's somehow problem on the side of the gstreamer library,
|
|
that provides bad responses, i'll try to use other languages \ libraries
|
|
|
|
but could also really use some help in figuring this out
|
|
|
|
fuller log:
|
|
#+begin_src
|
|
The value
|
|
NIL
|
|
is not of type
|
|
REAL
|
|
[Condition of type TYPE-ERROR]
|
|
|
|
Restarts:
|
|
0: [ABORT] abort thread (#<FOREIGN-THREAD tid=127655 "callback" RUNNING {10131D8033}>)
|
|
|
|
Backtrace:
|
|
0: (SB-KERNEL:TWO-ARG->= 0 NIL)
|
|
1: (GIR::MAP-C-ARRAY #<FUNCTION (LAMBDA (GIR::POS) :IN GIR::MEM-GET) {10131D9CBB}> #.(SB-SYS:INT-SAP #X7F1EC80EBBD8) #<GIR::C-ARRAY-TYPE {10131E0FA3}>)
|
|
Locals:
|
|
ARRAY = #.(SB-SYS:INT-SAP #X7F1EC80EBBD8)
|
|
C-ARRAY-TYPE = #<GIR::C-ARRAY-TYPE {10131E0FA3}>
|
|
FUNC = #<FUNCTION (LAMBDA (GIR::POS) :IN GIR::MEM-GET) {10131D9CBB}>
|
|
I = 0
|
|
LENGTH = NIL
|
|
PARAM-SIZE = 1
|
|
POS = #.(SB-SYS:INT-SAP #X7F1EC80EBBD8)
|
|
2: ((:METHOD GIR::MEM-GET (T GIR::POINTER-TYPE)) #.(SB-SYS:INT-SAP #X7F1F97F7FFF8) #<GIR::POINTER-TYPE {10131E0FF3}>) [fast-method]
|
|
Locals:
|
|
GIR::POS = #.(SB-SYS:INT-SAP #X7F1F97F7FFF8)
|
|
TYPE = #<GIR::POINTER-TYPE {10131E0FF3}>
|
|
3: (GIR.FIELD:GET #.(SB-SYS:INT-SAP #X7F1ECC0036F0) #<GIR:FIELD-INFO {10087D1133}>)
|
|
4: (PROCESS-FRAME #<GIR::STRUCT-INSTANCE {10131E02D3}>)
|
|
5: ((LAMBDA (APPSINK) :IN START-PROCESSING) #<GIR::OBJECT-INSTANCE {10131E01B3}>)
|
|
6: ((LAMBDA (GIR::CLOSURE RETURN GIR::N-VALUES GIR::PARAMS GIR::HINT GIR::DATA) :IN "/nix/store/hijk9y2lcv24s3a5kv56cyxqcn3frgs4-source-patched/src/signal.lisp") #.(SB-SYS:INT-SAP #X7F1FCC470B30) #.(SB-S..
|
|
7: ((LAMBDA (SB-ALIEN::ARGS-POINTER SB-ALIEN::RESULT-POINTER FUNCTION) :IN "/nix/store/hijk9y2lcv24s3a5kv56cyxqcn3frgs4-source-patched/src/signal.lisp") #<unavailable argument> #<unavailable argument> #<..
|
|
8: ((FLET SB-UNIX::BODY :IN SB-THREAD::RUN))
|
|
9: ((FLET "WITHOUT-INTERRUPTS-BODY-" :IN SB-THREAD::RUN))
|
|
10: ((FLET SB-UNIX::BODY :IN SB-THREAD::RUN))
|
|
11: ((FLET "WITHOUT-INTERRUPTS-BODY-" :IN SB-THREAD::RUN))
|
|
12: (SB-THREAD::RUN)
|
|
13: (SB-THREAD::ENTER-FOREIGN-CALLBACK 87 69885224809132 69885224809124)
|
|
14: ("foreign function: call_into_lisp_")
|
|
15: ("foreign function: funcall3")
|
|
16: ("foreign function: callback_wrapper_trampoline")
|
|
17: ("foreign function: #x501053AB")
|
|
--more--
|
|
|
|
#+end_src
|
|
|
|
the source:
|
|
|
|
( in the git : https://git.sunshine.industries/efim/scratch-screen-share/src/commit/73316581a056a20e51dc36eb96673acfee7d90b0/gstreamer/capturing-frames.lisp )
|
|
|
|
#+begin_src lisp
|
|
;(load (sb-ext:posix-getenv "ASDF"))
|
|
(asdf:load-system 'cl-cffi-gtk)
|
|
(asdf:load-system 'cl-gobject-introspection)
|
|
|
|
(defpackage :capturing-frames
|
|
(:use :cl))
|
|
|
|
(in-package :capturing-frames)
|
|
|
|
(defvar *gstreamer* (gir:require-namespace "Gst"))
|
|
;(defvar *gstapp* (gir:require-namespace "GstApp"))
|
|
|
|
(defun cast-with-videobox (&optional stream-num)
|
|
(gir:invoke (*gstreamer* 'init) '())
|
|
(let* ((pipeline (gir:invoke (*gstreamer* "Pipeline" 'new) "video-pipeline"))
|
|
(source (gir:invoke (*gstreamer* "ElementFactory" 'make) (if stream-num
|
|
"pipewiresrc"
|
|
"videotestsrc") "source"))
|
|
(convert (gir:invoke (*gstreamer* "ElementFactory" 'make) "videoconvert" "convert"))
|
|
(tee (gir:invoke (*gstreamer* "ElementFactory" 'make) "tee" "tee"))
|
|
(queue1 (gir:invoke (*gstreamer* "ElementFactory" 'make) "queue" "queue1"))
|
|
(queue2 (gir:invoke (*gstreamer* "ElementFactory" 'make) "queue" "queue2"))
|
|
(sink (gir:invoke (*gstreamer* "ElementFactory" 'make) "autovideosink" "sink"))
|
|
(appsink (gir:invoke (*gstreamer* "ElementFactory" 'make) "appsink" "appsink"))
|
|
)
|
|
(when stream-num
|
|
;; Set pipewiresrc properties, such as the path
|
|
(setf (gir:property source 'path) (format nil "~A" stream-num)))
|
|
|
|
;; Add and link elements in the pipeline
|
|
(gir:invoke (pipeline 'add) source)
|
|
(gir:invoke (pipeline 'add) convert)
|
|
(gir:invoke (pipeline 'add) tee)
|
|
(gir:invoke (pipeline 'add) queue1)
|
|
(gir:invoke (pipeline 'add) queue2)
|
|
(gir:invoke (pipeline 'add) sink)
|
|
(gir:invoke (pipeline 'add) appsink)
|
|
|
|
(gir:invoke (source 'link) convert)
|
|
(gir:invoke (convert 'link) tee)
|
|
(gir:invoke (tee 'link) queue1)
|
|
(gir:invoke (tee 'link) queue2)
|
|
(gir:invoke (queue1 'link) sink)
|
|
(gir:invoke (queue2 'link) appsink)
|
|
|
|
;; Configure appsink
|
|
(setf (gir:property appsink "emit-signals") t
|
|
(gir:property appsink "max-buffers") 1
|
|
(gir:property appsink "drop") t)
|
|
|
|
(gir:invoke (pipeline 'set-state) (gir:nget *gstreamer* "State" :playing))
|
|
|
|
(values pipeline appsink)))
|
|
|
|
(defun process-frame (sample)
|
|
(format t ">>> got to process sample with ~A~%" sample)
|
|
(let* ((buffer (gir:invoke (sample 'get-buffer)))
|
|
(info (gir:invoke (sample 'get-info)))
|
|
(caps (gir:invoke (sample 'get-caps)))
|
|
(caps-struct (gir:invoke (caps 'get-structure) 0))
|
|
(width (gir:invoke ((gir:invoke (caps-struct 'get-value) "width") 'get-int)))
|
|
(height (gir:invoke ((gir:invoke (caps-struct 'get-value) "height") 'get-int))))
|
|
|
|
(multiple-value-bind (found map-info) (gir:invoke (buffer 'map) (gir:nget *gstreamer* "MapFlags" :read))
|
|
(format t ">>>> the buffer has offset ~A~%" (gir:field buffer 'offset))
|
|
(format t ">>>> what's in struct? ~A~%" (gir:invoke (caps-struct 'to-string)))
|
|
(format t ">>>> the fields are width ~A, height ~A~%" width height)
|
|
(format t ">>>> and map result are ~A and ~A~%" found map-info)
|
|
(when found
|
|
(let ((data (gir:field map-info 'data))
|
|
(size (gir:field map-info 'size)))
|
|
(format t ">>>> and struct info? ~A~%" map-info)
|
|
(gir:invoke (buffer 'unmap) map-info)
|
|
(format t ">>>> and data has size ~A and data is ~A~%" size (or data "empty data")))))))
|
|
|
|
(defun start-processing (appsink)
|
|
(gir:connect appsink "new-sample"
|
|
(lambda (appsink)
|
|
(format t ">> got printed object ~A~%" appsink)
|
|
(format t ">> got printed class ~A~%" (gir:gir-class-of appsink))
|
|
(format t ">> trying to get method on sink ~A~%" (gir:nget appsink 'pull-sample))
|
|
(let ((sample (gir:invoke (appsink 'pull-sample))))
|
|
(when sample
|
|
(process-frame sample)))
|
|
|
|
|
|
(gir:nget *gstreamer* "FlowReturn" :ok))))
|
|
|
|
(defun try-run ()
|
|
(multiple-value-bind (pipeline appsink) (cast-with-videobox)
|
|
(format t "got appsink ~A" appsink)
|
|
(start-processing appsink)
|
|
(loop
|
|
(sleep 1))))
|
|
|
|
;(try-run)
|
|
#+end_src
|
|
|
|
*** submitted
|
|
https://github.com/andy128k/cl-gobject-introspection/issues/102
|
|
|
|
** what's next?
|
|
i could try another library for gir bindings.
|
|
not sure whether that would help much, but yeah
|
|
|
|
or i could just try to replicate this in python
|
|
or maybe in go?
|
|
|
|
** whelp gir2cl is also on top of cl-gobject-introspection
|
|
huh.
|
|
let's set up a directory with go stuff? i guess
|
|
|
|
trying in separate directory
|
|
with https://github.com/linuxdeepin/go-gir
|
|
|
|
but more realistically
|
|
https://pkg.go.dev/github.com/go-gst/go-gst
|
|
|
|
this is very much different, but yeah.
|
|
let's go with appsink example
|
|
https://pkg.go.dev/github.com/go-gst/go-gst@v1.1.0/examples/appsink
|
|
|
|
for the go library to pick up gstreamer from nix shell
|
|
https://discourse.nixos.org/t/flakes-how-to-make-pkg-config-work-with-nix-shell/16305
|
|
it has to somehow be exposed via pkg-config
|
|
|
|
yes, adding pkg-config to shell works, nice
|
|
|
|
* [2024-08-24 Sat]
|
|
|
|
** ok, continuing with go thing. into what do i want to decode stuff?
|
|
well, the reported default format is
|
|
sample info: video/x-raw, width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, format=(string)YV12, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
|
|
http://www.avisynth.nl/index.php/FAQ_YV12
|
|
|
|
but maybe i want something else?
|
|
|
|
let's read docs on appsrc
|
|
https://github.com/go-gst/go-gst/tree/v1.1.0/examples/appsrc
|
|
* [2024-12-08 Sun]
|
|
** well, i guess i will maybe return to this?
|
|
https://www.youtube.com/watch?v=Jn_09BeUEI0
|
|
here is gltransformation
|
|
this looks like what i'd really want, no?
|
|
** and this is documentation
|
|
https://gstreamer.freedesktop.org/documentation/opengl/index.html?gi-language=c
|
|
|
|
https://gstreamer.freedesktop.org/documentation/opengl/gltransformation.html?gi-language=c#gltransformation
|
|
why doesn't it work?
|
|
* [2024-12-21 Sat]
|
|
** wait. it works now? how and why
|
|
maybe because it's installed via apt?
|
|
yes. if i try to use
|
|
gst-launch-1.0 gltestsrc ! gltransformation scale-y=0.5 scale-x=0.5 ! glimagesink
|
|
in the nix shell i get
|
|
#+begin_src
|
|
Setting pipeline to PAUSED ...
|
|
Caught SIGSEGV
|
|
Spinning. Please run 'gdb gst-launch-1.0 242683' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.
|
|
#+end_src
|
|
** for now found article about other ways to transform
|
|
https://lubosz.wordpress.com/2014/06/16/transforming-video-on-the-gpu/
|
|
but that this opengl way is much faster
|
|
|
|
comparing with this:
|
|
https://gstreamer.freedesktop.org/documentation/videoconvertscale/videoscale.html?gi-language=c
|
|
gst-launch-1.0 -v videotestsrc ! videoscale ! autovideosink
|
|
|
|
|
|
gst-launch-1.0 -v videotestsrc ! videoscale ! video/x-raw,width=1000 ! autovideosink
|
|
** the ordinary error is
|
|
gst-launch-1.0 gltestsrc ! gltransformation rotation-z=45 ! glimagesink
|
|
Setting pipeline to PAUSED ...
|
|
Caught SIGSEGV
|
|
Spinning. Please run 'gdb gst-launch-1.0 7843' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.
|
|
|
|
** huh. trying previous nixpkgs releases
|
|
and in 23.05 the error is different
|
|
efim-nefedov@LLF33A87M:~/Documents/personal/scratch-screen-share$ gst-launch-1.0 gltestsrc ! gltransformation rotation-z=45 ! glimagesink
|
|
Setting pipeline to PAUSED ...
|
|
ERROR: from element /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink: Could not find any FBConfig's to use (check attributes?)
|
|
Additional debug info:
|
|
../ext/gl/gstglimagesink.c(1131): _ensure_gl_setup (): /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink
|
|
ERROR: pipeline doesn't want to preroll.
|
|
Failed to set pipeline to PAUSED.
|
|
Setting pipeline to NULL ...
|
|
Freeing pipeline ...
|
|
|
|
** ok, looking at the error the problem is in creating connection
|
|
WARN gldisplayegl gstgldisplay_egl.c:408:gst_gl_display_egl_from_gl_display:<gldisplayegl1> failed to get EGLDisplay from native display
|
|
|
|
but beats me if i know what needs to be done
|
|
claude.ai recommends adding LD_LIBRARY_PATH to gstreamer
|
|
so trying
|
|
#+begin_src nix
|
|
gstWrapper = pkgs.writeScriptBin "gst-launch-wrapped" ''
|
|
#!${pkgs.bash}/bin/bash
|
|
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH
|
|
exec ${pkgs.gst_all_1.gstreamer}/bin/gst-launch-1.0 "$@"
|
|
'';
|
|
#+end_src
|
|
|
|
but that fails
|
|
$ gst-launch-wrapped gltestsrc ! gltransformation rotation-z=45 ! glimagesink
|
|
/nix/store/p6k7xp1lsfmbdd731mlglrdj2d66mr82-bash-5.2p37/bin/bash: error while loading shared libraries: __vdso_gettimeofday: invalid mode for dlopen(): Invalid argument
|
|
|
|
* [2024-12-22 Sun]
|
|
|
|
** ok, let's try what
|
|
let's try default video out after i turn on the screen sharing from cl part
|
|
|
|
with code cleaned up i can start the share, that's nice
|
|
|
|
gst-launch-1.0 pipewiresrc path=90 ! videoconvert ! autovideosink
|
|
|
|
let's also try to add opengl processing
|
|
gst-launch-1.0 pipewiresrc path=90 ! videoconvert ! gltransformation rotation-z=45 ! glimagesink
|
|
|
|
so, this kind of works
|
|
gst-launch-1.0 pipewiresrc path=90 ! videoconvert ! capsfilter caps="video/x-raw,format=RGBA" ! glupload ! gltransformation rotation-z=45 ! glimagesink
|
|
|
|
even if the thing is small
|
|
|
|
** trying to figure out why gl things open a small window
|
|
*** one that works
|
|
GST_DEBUG=4,caps:5 gst-launch-1.0 pipewiresrc path=90 ! videoconvert ! autovideosink
|
|
has
|
|
deterministic way of creating a stream-id
|
|
0:00:00.032812186 44186 0x74b020000b70 INFO GST_EVENT gstevent.c:918:gst_event_new_caps: creating caps event video/x-raw, format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)0/1, max-framerate=(fraction)60/1
|
|
0:00:00.033493070 44186 0x74b020000b70 INFO GST_EVENT gstevent.c:918:gst_event_new_caps: creating caps event video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)0/1, max-framerate=(fraction)60/1, format=(string)YV12
|
|
|
|
*** and comparing with
|
|
GST_DEBUG=4,caps:5 gst-launch-1.0 pipewiresrc path=90 ! videoconvert ! capsfilter caps="video/x-raw,format=RGBA" ! glupload ! gltransformation rotation-z=45 ! glimagesink
|
|
(by recommendation of claude)
|
|
0:00:00.152714238 44533 0x784a70000b70 INFO GST_EVENT gstevent.c:918:gst_event_new_caps: creating caps event video/x-raw(memory:GLMemory), width=(int)1920, height=(int)1080, framerate=(fraction)0/1, max-framerate=(fraction)60/1, format=(string)RGBA, texture-target=(string)2D
|
|
0:00:00.152789062 44533 0x784a70000b70 INFO basetransform gstbasetransform.c:1326:gst_base_transform_setcaps:<glcolorbalance0> reuse caps
|
|
0:00:00.152821256 44533 0x784a70000b70 INFO GST_EVENT gstevent.c:918:gst_event_new_caps: creating caps event video/x-raw(memory:GLMemory), width=(int)1920, height=(int)1080, framerate=(fraction)0/1, max-framerate=(fraction)60/1, format=(string)RGBA, texture-target=(string)2D
|
|
|
|
well, i guess i can just open the window full size
|
|
|
|
*** ok! looks like it does stream full resolution.
|
|
now. i'd want to open the displaying pipeline programmatically
|
|
and then maybe try to dynamically change params
|
|
|
|
i guess even without a window embedding, just starting a window is good enough
|
|
but also, hm.
|
|
|
|
do i need any ui for the video window? i really don't think so.
|
|
maybe let's figure out the missing cursor
|
|
gst-launch-1.0 pipewiresrc path=90 ! videoconvert ! capsfilter caps="video/x-raw,format=RGBA" ! glupload ! gltransformation rotation-x=-15 ortho=True ! glimagesink
|
|
gst-launch-1.0 gltestsrc ! gltransformation rotation-x=-45 ortho=True ! glimagesink
|
|
|
|
|
|
** am i not getting into Create Session? after the Start?
|
|
https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.ScreenCast.html
|
|
(also note that here are cursor options)
|
|
|
|
** ok, let's try writing this in java then
|